[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:13:37 dist-test-slave-p401 krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:13:37 dist-test-slave-p401 krb5kdc[399](info): set up 2 sockets
Sep 24 04:13:37 dist-test-slave-p401 krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260924 04:13:38.896387 392 mini_kdc.cc:121] Time spent starting KDC: real 2.592s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260924 04:13:39.074651 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.178s user 0.002s sys 0.001s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260924 04:13:39.323093 392 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.156s user 0.000s sys 0.002s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/krb5kdc/test-user.keytab.
W20260924 04:13:40.304877 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.982s user 0.000s sys 0.002s
Sep 24 04:13:40 dist-test-slave-p401 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:13:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:13:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:13:40.399852 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40609
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40609
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:13:40.509562 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:13:40.509869 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:13:40.509919 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:13:40.514057 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:13:40.514125 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:13:40.514143 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:13:40.514159 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:13:40.514173 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:13:40.520311 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40609
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:13:40.521486 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:13:40.522596 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:13:40.532084 412 server_base.cc:1034] running on GCE node
W20260924 04:13:40.532065 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:13:40.532063 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:13:40.532095 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:13:40.533068 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:13:40.534204 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:13:40.535342 412 hybrid_clock.cc:648] HybridClock initialized: now 1790223220535289 us; error 81 us; skew 500 ppm
Sep 24 04:13:40 dist-test-slave-p401 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223220, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:13:40.543875 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:13:40.544976 412 webserver.cc:460] Webserver started at http://127.0.98.62:46161/ using document root <none> and password file <none>
I20260924 04:13:40.545596 412 fs_manager.cc:359] Metadata directory not provided
I20260924 04:13:40.545666 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:13:40.545876 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:13:40.548169 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/master-0/data/instance:
uuid: "985e56b54ffe4aa0863f9f2bb3f36996"
format_stamp: "Formatted at 2026-09-24 04:13:40 on dist-test-slave-p401"
server_key: "424a3d361fc1981cfa799ffa592e8e2f"
server_key_iv: "6ca4457178826ac88d523f483ba04f97"
server_key_version: "encryptionkey@0"
I20260924 04:13:40.548723 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/master-0/wal/instance:
uuid: "985e56b54ffe4aa0863f9f2bb3f36996"
format_stamp: "Formatted at 2026-09-24 04:13:40 on dist-test-slave-p401"
server_key: "424a3d361fc1981cfa799ffa592e8e2f"
server_key_iv: "6ca4457178826ac88d523f483ba04f97"
server_key_version: "encryptionkey@0"
I20260924 04:13:40.552661 412 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 04:13:40.555133 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:13:40.556456 412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:13:40.556571 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/master-0/wal
uuid: "985e56b54ffe4aa0863f9f2bb3f36996"
format_stamp: "Formatted at 2026-09-24 04:13:40 on dist-test-slave-p401"
server_key: "424a3d361fc1981cfa799ffa592e8e2f"
server_key_iv: "6ca4457178826ac88d523f483ba04f97"
server_key_version: "encryptionkey@0"
I20260924 04:13:40.556674 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:13:40.577865 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:13:40.581017 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:13:40.581208 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:13:40.588038 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40609
I20260924 04:13:40.588056 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40609 every 8 connection(s)
I20260924 04:13:40.588981 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:13:40.591714 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:13:40.596431 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 412
I20260924 04:13:40.596565 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/master-0/wal/instance
I20260924 04:13:40.597249 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 985e56b54ffe4aa0863f9f2bb3f36996: Bootstrap starting.
I20260924 04:13:40.597929 392 external_mini_cluster.cc:1428] Setting key 6860171c35ebb236d053b5d07304a405
I20260924 04:13:40.600005 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 985e56b54ffe4aa0863f9f2bb3f36996: Neither blocks nor log segments found. Creating new log.
I20260924 04:13:40.600822 480 log.cc:824] T 00000000000000000000000000000000 P 985e56b54ffe4aa0863f9f2bb3f36996: Log is configured to *not* fsync() on all Append() calls
I20260924 04:13:40.602733 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 985e56b54ffe4aa0863f9f2bb3f36996: No bootstrap required, opened a new log
I20260924 04:13:40.606532 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 985e56b54ffe4aa0863f9f2bb3f36996 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "985e56b54ffe4aa0863f9f2bb3f36996" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40609 } }
I20260924 04:13:40.606758 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 985e56b54ffe4aa0863f9f2bb3f36996 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:13:40.606808 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 985e56b54ffe4aa0863f9f2bb3f36996 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 985e56b54ffe4aa0863f9f2bb3f36996, State: Initialized, Role: FOLLOWER
I20260924 04:13:40.607257 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 985e56b54ffe4aa0863f9f2bb3f36996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "985e56b54ffe4aa0863f9f2bb3f36996" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40609 } }
I20260924 04:13:40.607414 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 985e56b54ffe4aa0863f9f2bb3f36996 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:13:40.607479 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 985e56b54ffe4aa0863f9f2bb3f36996 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:13:40.607570 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 985e56b54ffe4aa0863f9f2bb3f36996 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:13:40.608871 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 985e56b54ffe4aa0863f9f2bb3f36996 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "985e56b54ffe4aa0863f9f2bb3f36996" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40609 } }
I20260924 04:13:40.609187 480 leader_election.cc:302] T 00000000000000000000000000000000 P 985e56b54ffe4aa0863f9f2bb3f36996 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 985e56b54ffe4aa0863f9f2bb3f36996; no voters:
I20260924 04:13:40.609469 480 leader_election.cc:288] T 00000000000000000000000000000000 P 985e56b54ffe4aa0863f9f2bb3f36996 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:13:40.609580 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 985e56b54ffe4aa0863f9f2bb3f36996 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:13:40.609771 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 985e56b54ffe4aa0863f9f2bb3f36996 [term 1 LEADER]: Becoming Leader. State: Replica: 985e56b54ffe4aa0863f9f2bb3f36996, State: Running, Role: LEADER
I20260924 04:13:40.610080 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 985e56b54ffe4aa0863f9f2bb3f36996 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "985e56b54ffe4aa0863f9f2bb3f36996" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40609 } }
I20260924 04:13:40.610607 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 985e56b54ffe4aa0863f9f2bb3f36996 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:13:40.611295 485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 985e56b54ffe4aa0863f9f2bb3f36996 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 985e56b54ffe4aa0863f9f2bb3f36996. Latest consensus state: current_term: 1 leader_uuid: "985e56b54ffe4aa0863f9f2bb3f36996" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "985e56b54ffe4aa0863f9f2bb3f36996" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40609 } } }
I20260924 04:13:40.611415 485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 985e56b54ffe4aa0863f9f2bb3f36996 [sys.catalog]: This master's current role is: LEADER
I20260924 04:13:40.611709 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 985e56b54ffe4aa0863f9f2bb3f36996 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "985e56b54ffe4aa0863f9f2bb3f36996" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "985e56b54ffe4aa0863f9f2bb3f36996" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40609 } } }
I20260924 04:13:40.611804 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 985e56b54ffe4aa0863f9f2bb3f36996 [sys.catalog]: This master's current role is: LEADER
I20260924 04:13:40.611788 492 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:13:40.615162 492 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 04:13:40 dist-test-slave-p401 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:13:40.621500 492 catalog_manager.cc:1347] Generated new cluster ID: c07fde625c724d168d61587212870ad6
I20260924 04:13:40.621570 492 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:13:40.624766 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:40.600945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36636 (local address 127.0.98.62:40609)
0924 04:13:40.601389 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:40.601409 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:40.602663 (+ 1254us) server_negotiation.cc:407] Connection header received
0924 04:13:40.606157 (+ 3494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:40.606177 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:40.606520 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:40.606951 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:40.609442 (+ 2491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:40.611110 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:40.612993 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:40.613697 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:40.615285 (+ 1588us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:40.618640 (+ 3355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:13:40.618645 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:13:40.618658 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:13:40.618685 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:13:40.620698 (+ 2013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:40.621365 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:40.621371 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:40.621377 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:40.621546 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:40.622003 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:40.622006 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:40.622008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:40.622554 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:13:40.622774 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:40.623423 (+ 649us) server_negotiation.cc:299] Negotiation successful
0924 04:13:40.623812 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":130,"threads_started":1}
I20260924 04:13:40.631739 492 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:13:40.633013 492 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:13:40.639544 492 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 985e56b54ffe4aa0863f9f2bb3f36996: Generated new TSK 0
I20260924 04:13:40.640406 492 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:13:40.669812 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40609
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:13:40.783476 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:13:40.783761 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:13:40.783807 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:13:40.787999 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:13:40.788079 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:13:40.788143 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:13:40.793591 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40609
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:13:40.794638 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:13:40.795704 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:13:40.805001 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:13:40.804983 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:13:40.804975 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:13:40.805209 506 server_base.cc:1034] running on GCE node
I20260924 04:13:40.805716 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:13:40.806352 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:13:40.807510 506 hybrid_clock.cc:648] HybridClock initialized: now 1790223220807487 us; error 33 us; skew 500 ppm
Sep 24 04:13:40 dist-test-slave-p401 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223220, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:13:40.810122 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:13:40.811075 506 webserver.cc:460] Webserver started at http://127.0.98.1:37613/ using document root <none> and password file <none>
I20260924 04:13:40.811672 506 fs_manager.cc:359] Metadata directory not provided
I20260924 04:13:40.811734 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:13:40.811893 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:13:40.813886 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/ts-0/data/instance:
uuid: "c8227857c8104928a201e1dcf5891f9d"
format_stamp: "Formatted at 2026-09-24 04:13:40 on dist-test-slave-p401"
server_key: "7b88541f7d4cb8058baca6980548f0a5"
server_key_iv: "c01c4c3da86dbb9f3a812b65dfe8657d"
server_key_version: "encryptionkey@0"
I20260924 04:13:40.814419 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c8227857c8104928a201e1dcf5891f9d"
format_stamp: "Formatted at 2026-09-24 04:13:40 on dist-test-slave-p401"
server_key: "7b88541f7d4cb8058baca6980548f0a5"
server_key_iv: "c01c4c3da86dbb9f3a812b65dfe8657d"
server_key_version: "encryptionkey@0"
I20260924 04:13:40.818065 506 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 04:13:40.820375 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:13:40.821626 506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:13:40.821733 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/ts-0/wal
uuid: "c8227857c8104928a201e1dcf5891f9d"
format_stamp: "Formatted at 2026-09-24 04:13:40 on dist-test-slave-p401"
server_key: "7b88541f7d4cb8058baca6980548f0a5"
server_key_iv: "c01c4c3da86dbb9f3a812b65dfe8657d"
server_key_version: "encryptionkey@0"
I20260924 04:13:40.821822 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:13:40.854379 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:13:40.857702 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:13:40.857888 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:13:40.858471 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:13:40.859338 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:13:40.859387 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:13:40.859450 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:13:40.859475 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:13:40.868661 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44615
I20260924 04:13:40.868687 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44615 every 8 connection(s)
I20260924 04:13:40.869560 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:13:40.876391 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 506
I20260924 04:13:40.876487 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:13:40.876700 392 external_mini_cluster.cc:1428] Setting key 51a27e355766922fa1868cb22f62da8f
Sep 24 04:13:40 dist-test-slave-p401 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223220, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:13:40.882839 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:40.871337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34187 (local address 127.0.98.62:40609)
0924 04:13:40.871490 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:40.871494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:40.872409 (+ 915us) server_negotiation.cc:407] Connection header received
0924 04:13:40.873700 (+ 1291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:40.873703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:40.873765 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:40.873835 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:40.875006 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:40.875567 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:40.876722 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:40.876982 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:40.877072 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:40.879429 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:13:40.879432 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:13:40.879434 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:13:40.879458 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:13:40.880826 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:40.881323 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:40.881325 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:40.881327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:40.881404 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:40.881734 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:40.881737 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:40.881739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:40.881919 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:13:40.882001 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:40.882529 (+ 528us) server_negotiation.cc:299] Negotiation successful
0924 04:13:40.882666 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 04:13:40.883733 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:40.871774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40609 (local address 127.0.98.1:34187)
0924 04:13:40.872235 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:40.872281 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:40.873490 (+ 1209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:40.874012 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:40.874018 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:40.874376 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:13:40.874860 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:40.874874 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:40.875682 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:40.875685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:40.876605 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:40.876615 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:40.877123 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:40.877127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:40.877291 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:40.877828 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:13:40.877845 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:13:40.879256 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:13:40.880937 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:40.880943 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:40.880961 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:40.881223 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:40.881487 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:40.881490 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:40.881492 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:40.881640 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:40.882018 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:13:40.882026 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:13:40.882299 (+ 273us) client_negotiation.cc:769] Sending connection context
0924 04:13:40.882472 (+ 173us) client_negotiation.cc:240] Negotiation successful
0924 04:13:40.882823 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":117,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:13:40.884943 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40609
I20260924 04:13:40.885169 635 heartbeater.cc:464] Registering TS with master...
I20260924 04:13:40.885653 635 heartbeater.cc:511] Master 127.0.98.62:40609 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:13:40.887290 444 ts_manager.cc:194] Registered new tserver with Master: c8227857c8104928a201e1dcf5891f9d (127.0.98.1:44615)
I20260924 04:13:40.888402 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34187
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:13:40.902439 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40609
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:13:41.010982 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:13:41.011302 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:13:41.011385 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:13:41.015588 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:13:41.015651 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:13:41.015722 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:13:41.021176 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40609
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:13:41.022248 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:13:41.023280 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:13:41.033655 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:13:41.033703 640 server_base.cc:1034] running on GCE node
W20260924 04:13:41.033655 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:13:41.033780 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:13:41.034334 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:13:41.034989 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:13:41.036142 640 hybrid_clock.cc:648] HybridClock initialized: now 1790223221036111 us; error 44 us; skew 500 ppm
Sep 24 04:13:41 dist-test-slave-p401 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223221, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:13:41.038765 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:13:41.039641 640 webserver.cc:460] Webserver started at http://127.0.98.2:43449/ using document root <none> and password file <none>
I20260924 04:13:41.040177 640 fs_manager.cc:359] Metadata directory not provided
I20260924 04:13:41.040241 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:13:41.040447 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:13:41.042435 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/ts-1/data/instance:
uuid: "443193207564459e95587294351e767e"
format_stamp: "Formatted at 2026-09-24 04:13:41 on dist-test-slave-p401"
server_key: "515ffbf0eea50f83ad534724c0c17fb4"
server_key_iv: "8b1358f5f39be783f97280ef33da6637"
server_key_version: "encryptionkey@0"
I20260924 04:13:41.042981 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/ts-1/wal/instance:
uuid: "443193207564459e95587294351e767e"
format_stamp: "Formatted at 2026-09-24 04:13:41 on dist-test-slave-p401"
server_key: "515ffbf0eea50f83ad534724c0c17fb4"
server_key_iv: "8b1358f5f39be783f97280ef33da6637"
server_key_version: "encryptionkey@0"
I20260924 04:13:41.046590 640 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 04:13:41.048915 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:13:41.050058 640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:13:41.050156 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/ts-1/wal
uuid: "443193207564459e95587294351e767e"
format_stamp: "Formatted at 2026-09-24 04:13:41 on dist-test-slave-p401"
server_key: "515ffbf0eea50f83ad534724c0c17fb4"
server_key_iv: "8b1358f5f39be783f97280ef33da6637"
server_key_version: "encryptionkey@0"
I20260924 04:13:41.050246 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:13:41.065587 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:13:41.068394 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:13:41.068564 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:13:41.069080 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:13:41.069859 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:13:41.069902 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:13:41.069940 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:13:41.069963 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:13:41.078935 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35991
I20260924 04:13:41.078958 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35991 every 8 connection(s)
I20260924 04:13:41.079883 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:13:41.088572 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 640
I20260924 04:13:41.088671 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/ts-1/wal/instance
Sep 24 04:13:41 dist-test-slave-p401 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223221, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:13:41.088923 392 external_mini_cluster.cc:1428] Setting key 7b75d1dac48f25a987796d0eeaeb559e
I20260924 04:13:41.092607 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:41.081651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46337 (local address 127.0.98.62:40609)
0924 04:13:41.081781 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:41.081785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:41.082585 (+ 800us) server_negotiation.cc:407] Connection header received
0924 04:13:41.083884 (+ 1299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:41.083887 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:41.083942 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:41.084011 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:41.085149 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:41.085656 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:41.086701 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:41.086915 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:41.086980 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:41.089320 (+ 2340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:13:41.089323 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:13:41.089325 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:13:41.089348 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:13:41.090678 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:41.091155 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:41.091158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:41.091159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:41.091256 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:41.091579 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:41.091582 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:41.091584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:41.091756 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:13:41.091836 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:41.092327 (+ 491us) server_negotiation.cc:299] Negotiation successful
0924 04:13:41.092451 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 04:13:41.093420 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:41.082020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40609 (local address 127.0.98.2:46337)
0924 04:13:41.082440 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:41.082482 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:41.083687 (+ 1205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:41.084176 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:41.084182 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:41.084548 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:13:41.085029 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:41.085042 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:41.085741 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:41.085744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:41.086610 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:41.086618 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:41.087022 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:41.087024 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:41.087161 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:41.087748 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:13:41.087764 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:13:41.089158 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:13:41.090782 (+ 1624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:41.090788 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:41.090804 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:41.091066 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:41.091334 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:41.091337 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:41.091339 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:41.091492 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:41.091849 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:13:41.091855 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:13:41.092110 (+ 255us) client_negotiation.cc:769] Sending connection context
0924 04:13:41.092277 (+ 167us) client_negotiation.cc:240] Negotiation successful
0924 04:13:41.092615 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":102,"threads_started":1}
I20260924 04:13:41.094480 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40609
I20260924 04:13:41.094753 769 heartbeater.cc:464] Registering TS with master...
I20260924 04:13:41.095372 769 heartbeater.cc:511] Master 127.0.98.62:40609 requested a full tablet report, sending...
I20260924 04:13:41.096493 444 ts_manager.cc:194] Registered new tserver with Master: 443193207564459e95587294351e767e (127.0.98.2:35991)
I20260924 04:13:41.097263 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46337
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:13:41.114379 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40609
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:13:41.223565 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:13:41.223846 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:13:41.223891 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:13:41.228034 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:13:41.228093 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:13:41.228154 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:13:41.233553 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40609
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:13:41.234524 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:13:41.235519 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:13:41.244920 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:13:41.244999 774 server_base.cc:1034] running on GCE node
W20260924 04:13:41.245050 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:13:41.244947 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:13:41.245532 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:13:41.246173 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:13:41.247293 774 hybrid_clock.cc:648] HybridClock initialized: now 1790223221247270 us; error 33 us; skew 500 ppm
Sep 24 04:13:41 dist-test-slave-p401 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223221, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:13:41.249933 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:13:41.250727 774 webserver.cc:460] Webserver started at http://127.0.98.3:42567/ using document root <none> and password file <none>
I20260924 04:13:41.251271 774 fs_manager.cc:359] Metadata directory not provided
I20260924 04:13:41.251343 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:13:41.251523 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:13:41.253518 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/ts-2/data/instance:
uuid: "14945f16d34e4807b43f9410414719ab"
format_stamp: "Formatted at 2026-09-24 04:13:41 on dist-test-slave-p401"
server_key: "132ed3e7fef3e93e5a808aba4b792036"
server_key_iv: "48c341fd27fcaf14a93d3a2b13d91726"
server_key_version: "encryptionkey@0"
I20260924 04:13:41.254076 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/ts-2/wal/instance:
uuid: "14945f16d34e4807b43f9410414719ab"
format_stamp: "Formatted at 2026-09-24 04:13:41 on dist-test-slave-p401"
server_key: "132ed3e7fef3e93e5a808aba4b792036"
server_key_iv: "48c341fd27fcaf14a93d3a2b13d91726"
server_key_version: "encryptionkey@0"
I20260924 04:13:41.257563 774 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:13:41.259903 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:13:41.261153 774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:13:41.261253 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/ts-2/wal
uuid: "14945f16d34e4807b43f9410414719ab"
format_stamp: "Formatted at 2026-09-24 04:13:41 on dist-test-slave-p401"
server_key: "132ed3e7fef3e93e5a808aba4b792036"
server_key_iv: "48c341fd27fcaf14a93d3a2b13d91726"
server_key_version: "encryptionkey@0"
I20260924 04:13:41.261351 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:13:41.281313 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:13:41.284199 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:13:41.284366 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:13:41.284945 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:13:41.285739 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:13:41.285784 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:13:41.285828 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:13:41.285856 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:13:41.294513 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42971
I20260924 04:13:41.294551 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42971 every 8 connection(s)
I20260924 04:13:41.295555 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:13:41.300312 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 774
I20260924 04:13:41.300407 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790223216265354-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:13:41.300611 392 external_mini_cluster.cc:1428] Setting key 3904f9cdd4d9c31470aaa09061530a1c
Sep 24 04:13:41 dist-test-slave-p401 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223221, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:13:41.308388 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:41.297305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33741 (local address 127.0.98.62:40609)
0924 04:13:41.297437 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:41.297441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:41.298265 (+ 824us) server_negotiation.cc:407] Connection header received
0924 04:13:41.299581 (+ 1316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:41.299584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:41.299638 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:41.299710 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:41.300908 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:41.301476 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:41.302961 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:41.303168 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:41.303272 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:41.305444 (+ 2172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:13:41.305448 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:13:41.305450 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:13:41.305473 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:13:41.306539 (+ 1066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:41.306982 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:41.306985 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:41.306986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:41.307054 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:41.307390 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:41.307393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:41.307395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:41.307572 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:13:41.307653 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:41.308124 (+ 471us) server_negotiation.cc:299] Negotiation successful
0924 04:13:41.308236 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 04:13:41.309123 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:41.297694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40609 (local address 127.0.98.3:33741)
0924 04:13:41.298123 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:41.298164 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:41.299378 (+ 1214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:41.299886 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:41.299892 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:41.300261 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:13:41.300780 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:41.300795 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:41.301590 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:41.301592 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:41.302841 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:41.302850 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:41.303319 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:41.303323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:41.303484 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:41.303990 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:13:41.304006 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:13:41.305292 (+ 1286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:13:41.306634 (+ 1342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:41.306640 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:41.306654 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:41.306887 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:41.307129 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:41.307132 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:41.307134 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:41.307309 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:41.307664 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:13:41.307670 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:13:41.307894 (+ 224us) client_negotiation.cc:769] Sending connection context
0924 04:13:41.308052 (+ 158us) client_negotiation.cc:240] Negotiation successful
0924 04:13:41.308348 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":110,"threads_started":1}
I20260924 04:13:41.310074 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40609
I20260924 04:13:41.310279 903 heartbeater.cc:464] Registering TS with master...
I20260924 04:13:41.310693 903 heartbeater.cc:511] Master 127.0.98.62:40609 requested a full tablet report, sending...
I20260924 04:13:41.311614 444 ts_manager.cc:194] Registered new tserver with Master: 14945f16d34e4807b43f9410414719ab (127.0.98.3:42971)
I20260924 04:13:41.312130 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33741
I20260924 04:13:41.314461 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 04:13:41 dist-test-slave-p401 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223221, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 04:13:41 dist-test-slave-p401 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223221, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:13:41.339408 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:41.330648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47236 (local address 127.0.98.1:44615)
0924 04:13:41.330956 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:41.330973 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:41.331000 (+ 27us) server_negotiation.cc:407] Connection header received
0924 04:13:41.331062 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:41.331067 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:41.331325 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:41.331471 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:41.332046 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:41.332827 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:41.333624 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:41.333997 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:41.334092 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:41.336447 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:13:41.336457 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:13:41.336472 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:13:41.336496 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:13:41.337873 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:41.338273 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:41.338278 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:41.338279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:41.338351 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:41.338554 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:41.338556 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:41.338557 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:41.338774 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:13:41.338940 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:41.339052 (+ 112us) server_negotiation.cc:299] Negotiation successful
0924 04:13:41.339177 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":126,"threads_started":1}
W20260924 04:13:41.340410 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47236
Sep 24 04:13:41 dist-test-slave-p401 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 04:13:41 dist-test-slave-p401 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:13:41.363464 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:41.356177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47252 (local address 127.0.98.1:44615)
0924 04:13:41.356327 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:41.356331 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:41.356415 (+ 84us) server_negotiation.cc:407] Connection header received
0924 04:13:41.356533 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:41.356535 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:41.356577 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:41.356635 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:41.357243 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:41.357743 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:41.358479 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:41.358703 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:41.358775 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:41.361002 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:13:41.361005 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:13:41.361007 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:13:41.361031 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:13:41.362168 (+ 1137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:41.362505 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:41.362508 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:41.362509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:41.362571 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:41.362776 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:41.362778 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:41.362779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:41.362922 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:13:41.363010 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:41.363180 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 04:13:41.363302 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 04:13:41.364392 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 506
I20260924 04:13:41.369585 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 640
I20260924 04:13:41.375638 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 774
I20260924 04:13:41.380079 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 412
2026-09-24T04:13:41Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (5105 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:13:41 dist-test-slave-p401 krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:13:41 dist-test-slave-p401 krb5kdc[917](info): set up 2 sockets
Sep 24 04:13:41 dist-test-slave-p401 krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260924 04:13:42.330643 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/krb5kdc/test-user.keytab.
Sep 24 04:13:42 dist-test-slave-p401 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:13:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:13:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:13:42.448103 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38797
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38797
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:13:42.553671 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:13:42.553944 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:13:42.553996 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:13:42.557933 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:13:42.557987 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:13:42.558013 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:13:42.558032 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:13:42.558053 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:13:42.563212 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:13:42.564250 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:13:42.565356 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:13:42.573416 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:13:42.573417 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:13:42.573554 930 server_base.cc:1034] running on GCE node
W20260924 04:13:42.573416 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:13:42.574237 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:13:42.575281 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:13:42.576443 930 hybrid_clock.cc:648] HybridClock initialized: now 1790223222576416 us; error 40 us; skew 500 ppm
Sep 24 04:13:42 dist-test-slave-p401 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223222, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:13:42.579038 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:13:42.579900 930 webserver.cc:460] Webserver started at http://127.0.98.62:41211/ using document root <none> and password file <none>
I20260924 04:13:42.580412 930 fs_manager.cc:359] Metadata directory not provided
I20260924 04:13:42.580476 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:13:42.580648 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:13:42.582669 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/master-0/data/instance:
uuid: "d78e582039214ea2b8d739e0ab7f7249"
format_stamp: "Formatted at 2026-09-24 04:13:42 on dist-test-slave-p401"
server_key: "0d9211cd7c48ed586526578ab1d6d342"
server_key_iv: "7a97262a96c9ab9e7a9c4c9fa05292ce"
server_key_version: "encryptionkey@0"
I20260924 04:13:42.583199 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/master-0/wal/instance:
uuid: "d78e582039214ea2b8d739e0ab7f7249"
format_stamp: "Formatted at 2026-09-24 04:13:42 on dist-test-slave-p401"
server_key: "0d9211cd7c48ed586526578ab1d6d342"
server_key_iv: "7a97262a96c9ab9e7a9c4c9fa05292ce"
server_key_version: "encryptionkey@0"
I20260924 04:13:42.586659 930 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 04:13:42.589002 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:13:42.590158 930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:13:42.590268 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/master-0/wal
uuid: "d78e582039214ea2b8d739e0ab7f7249"
format_stamp: "Formatted at 2026-09-24 04:13:42 on dist-test-slave-p401"
server_key: "0d9211cd7c48ed586526578ab1d6d342"
server_key_iv: "7a97262a96c9ab9e7a9c4c9fa05292ce"
server_key_version: "encryptionkey@0"
I20260924 04:13:42.590356 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:13:42.616742 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:13:42.619941 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:13:42.620117 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:13:42.626669 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38797
I20260924 04:13:42.626684 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38797 every 8 connection(s)
I20260924 04:13:42.627686 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:13:42.630309 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:13:42.633520 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 930
I20260924 04:13:42.633623 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/master-0/wal/instance
I20260924 04:13:42.633833 392 external_mini_cluster.cc:1428] Setting key 27b83be75662c7724f0c7da09bfcf968
I20260924 04:13:42.635943 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d78e582039214ea2b8d739e0ab7f7249: Bootstrap starting.
I20260924 04:13:42.638233 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d78e582039214ea2b8d739e0ab7f7249: Neither blocks nor log segments found. Creating new log.
I20260924 04:13:42.638985 998 log.cc:824] T 00000000000000000000000000000000 P d78e582039214ea2b8d739e0ab7f7249: Log is configured to *not* fsync() on all Append() calls
I20260924 04:13:42.640849 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d78e582039214ea2b8d739e0ab7f7249: No bootstrap required, opened a new log
Sep 24 04:13:42 dist-test-slave-p401 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:13:42.644053 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P d78e582039214ea2b8d739e0ab7f7249 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d78e582039214ea2b8d739e0ab7f7249" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38797 } }
I20260924 04:13:42.644321 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P d78e582039214ea2b8d739e0ab7f7249 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:13:42.644378 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P d78e582039214ea2b8d739e0ab7f7249 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d78e582039214ea2b8d739e0ab7f7249, State: Initialized, Role: FOLLOWER
I20260924 04:13:42.644804 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P d78e582039214ea2b8d739e0ab7f7249 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d78e582039214ea2b8d739e0ab7f7249" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38797 } }
I20260924 04:13:42.644927 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P d78e582039214ea2b8d739e0ab7f7249 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:13:42.644989 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P d78e582039214ea2b8d739e0ab7f7249 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:13:42.645079 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d78e582039214ea2b8d739e0ab7f7249 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:13:42.645900 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P d78e582039214ea2b8d739e0ab7f7249 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d78e582039214ea2b8d739e0ab7f7249" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38797 } }
I20260924 04:13:42.646209 998 leader_election.cc:302] T 00000000000000000000000000000000 P d78e582039214ea2b8d739e0ab7f7249 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d78e582039214ea2b8d739e0ab7f7249; no voters:
I20260924 04:13:42.646495 998 leader_election.cc:288] T 00000000000000000000000000000000 P d78e582039214ea2b8d739e0ab7f7249 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:13:42.646593 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d78e582039214ea2b8d739e0ab7f7249 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:13:42.646823 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P d78e582039214ea2b8d739e0ab7f7249 [term 1 LEADER]: Becoming Leader. State: Replica: d78e582039214ea2b8d739e0ab7f7249, State: Running, Role: LEADER
I20260924 04:13:42.647192 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P d78e582039214ea2b8d739e0ab7f7249 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d78e582039214ea2b8d739e0ab7f7249" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38797 } }
I20260924 04:13:42.647836 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P d78e582039214ea2b8d739e0ab7f7249 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:13:42.648080 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:42.635093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38462 (local address 127.0.98.62:38797)
0924 04:13:42.635512 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:42.635528 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:42.635565 (+ 37us) server_negotiation.cc:407] Connection header received
0924 04:13:42.636201 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:42.636219 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:42.636557 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:42.636824 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:42.637636 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:42.638630 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:42.639667 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:42.640121 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:42.640269 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:42.642793 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:13:42.642798 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:13:42.642811 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:13:42.642837 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:13:42.645000 (+ 2163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:42.645403 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:42.645409 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:42.645415 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:42.645525 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:42.645766 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:42.645769 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:42.645771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:42.646204 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:13:42.646370 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:42.646768 (+ 398us) server_negotiation.cc:299] Negotiation successful
0924 04:13:42.647086 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":143,"threads_started":1}
I20260924 04:13:42.648267 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P d78e582039214ea2b8d739e0ab7f7249 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d78e582039214ea2b8d739e0ab7f7249. Latest consensus state: current_term: 1 leader_uuid: "d78e582039214ea2b8d739e0ab7f7249" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d78e582039214ea2b8d739e0ab7f7249" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38797 } } }
I20260924 04:13:42.648404 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P d78e582039214ea2b8d739e0ab7f7249 [sys.catalog]: This master's current role is: LEADER
I20260924 04:13:42.648778 1007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:13:42.648746 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P d78e582039214ea2b8d739e0ab7f7249 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d78e582039214ea2b8d739e0ab7f7249" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d78e582039214ea2b8d739e0ab7f7249" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38797 } } }
I20260924 04:13:42.648943 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P d78e582039214ea2b8d739e0ab7f7249 [sys.catalog]: This master's current role is: LEADER
I20260924 04:13:42.651949 1007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:13:42.657157 1007 catalog_manager.cc:1347] Generated new cluster ID: e403c0959fdf45639ead90a97c860175
I20260924 04:13:42.657224 1007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:13:42.669202 1007 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:13:42.670140 1007 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:13:42.678721 1007 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d78e582039214ea2b8d739e0ab7f7249: Generated new TSK 0
I20260924 04:13:42.679603 1007 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:13:42.710331 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38797
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:13:42.818614 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:13:42.818861 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:13:42.818910 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:13:42.822835 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:13:42.822898 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:13:42.822954 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:13:42.828088 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38797
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:13:42.829049 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:13:42.830005 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:13:42.839244 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:13:42.839263 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:13:42.839352 1024 server_base.cc:1034] running on GCE node
W20260924 04:13:42.839248 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:13:42.839994 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:13:42.840691 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:13:42.841858 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790223222841844 us; error 26 us; skew 500 ppm
Sep 24 04:13:42 dist-test-slave-p401 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223222, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:13:42.844573 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:13:42.845465 1024 webserver.cc:460] Webserver started at http://127.0.98.1:44657/ using document root <none> and password file <none>
I20260924 04:13:42.845997 1024 fs_manager.cc:359] Metadata directory not provided
I20260924 04:13:42.846061 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:13:42.846264 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:13:42.848393 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/ts-0/data/instance:
uuid: "0101749bea08485a9fd7d03240f8d45a"
format_stamp: "Formatted at 2026-09-24 04:13:42 on dist-test-slave-p401"
server_key: "6bb6983ed71f8513dca84b66ddb408ef"
server_key_iv: "6ffd23694c8feb13b8626e93786b540e"
server_key_version: "encryptionkey@0"
I20260924 04:13:42.848929 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0101749bea08485a9fd7d03240f8d45a"
format_stamp: "Formatted at 2026-09-24 04:13:42 on dist-test-slave-p401"
server_key: "6bb6983ed71f8513dca84b66ddb408ef"
server_key_iv: "6ffd23694c8feb13b8626e93786b540e"
server_key_version: "encryptionkey@0"
I20260924 04:13:42.852671 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:13:42.855159 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:13:42.856429 1024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:13:42.856546 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/ts-0/wal
uuid: "0101749bea08485a9fd7d03240f8d45a"
format_stamp: "Formatted at 2026-09-24 04:13:42 on dist-test-slave-p401"
server_key: "6bb6983ed71f8513dca84b66ddb408ef"
server_key_iv: "6ffd23694c8feb13b8626e93786b540e"
server_key_version: "encryptionkey@0"
I20260924 04:13:42.856652 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:13:42.878064 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:13:42.881233 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:13:42.881445 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:13:42.882047 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:13:42.882887 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:13:42.882934 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:13:42.882987 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:13:42.883014 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:13:42.891620 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44969
I20260924 04:13:42.891654 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44969 every 8 connection(s)
I20260924 04:13:42.892510 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:13:42.896075 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 1024
I20260924 04:13:42.896162 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:13:42.896395 392 external_mini_cluster.cc:1428] Setting key 419cb214fd35af39f682614cf79e22c5
Sep 24 04:13:42 dist-test-slave-p401 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223222, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:13:42.906234 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:42.894217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39109 (local address 127.0.98.62:38797)
0924 04:13:42.894348 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:42.894352 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:42.895307 (+ 955us) server_negotiation.cc:407] Connection header received
0924 04:13:42.896791 (+ 1484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:42.896796 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:42.896857 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:42.896939 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:42.898179 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:42.898719 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:42.899975 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:42.900343 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:42.900452 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:42.902860 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:13:42.902863 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:13:42.902865 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:13:42.902904 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:13:42.904168 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:42.904758 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:42.904761 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:42.904762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:42.904835 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:42.905159 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:42.905162 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:42.905164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:42.905335 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:13:42.905416 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:42.905955 (+ 539us) server_negotiation.cc:299] Negotiation successful
0924 04:13:42.906074 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 04:13:42.907137 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:42.894611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38797 (local address 127.0.98.1:39109)
0924 04:13:42.895099 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:42.895147 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:42.896571 (+ 1424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:42.897126 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:42.897133 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:42.897522 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:13:42.898029 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:42.898045 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:42.898858 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:42.898862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:42.899852 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:42.899862 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:42.900483 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:42.900485 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:42.900625 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:42.901163 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:13:42.901178 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:13:42.902688 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:13:42.904284 (+ 1596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:42.904291 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:42.904308 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:42.904655 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:42.904932 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:42.904935 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:42.904937 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:42.905075 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:42.905434 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:13:42.905440 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:13:42.905718 (+ 278us) client_negotiation.cc:769] Sending connection context
0924 04:13:42.905891 (+ 173us) client_negotiation.cc:240] Negotiation successful
0924 04:13:42.906214 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":110,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:13:42.908401 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38797
I20260924 04:13:42.908643 1153 heartbeater.cc:464] Registering TS with master...
I20260924 04:13:42.909101 1153 heartbeater.cc:511] Master 127.0.98.62:38797 requested a full tablet report, sending...
I20260924 04:13:42.910488 962 ts_manager.cc:194] Registered new tserver with Master: 0101749bea08485a9fd7d03240f8d45a (127.0.98.1:44969)
I20260924 04:13:42.911633 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39109
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:13:42.921612 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38797
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:13:43.028637 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:13:43.028908 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:13:43.028960 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:13:43.032969 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:13:43.033027 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:13:43.033095 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:13:43.038482 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38797
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:13:43.039482 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:13:43.040444 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:13:43.049705 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:13:43.049855 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:13:43.049715 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:13:43.049961 1158 server_base.cc:1034] running on GCE node
I20260924 04:13:43.050467 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:13:43.051108 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:13:43.052251 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790223223052221 us; error 42 us; skew 500 ppm
Sep 24 04:13:43 dist-test-slave-p401 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223223, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:13:43.054850 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:13:43.055755 1158 webserver.cc:460] Webserver started at http://127.0.98.2:38893/ using document root <none> and password file <none>
I20260924 04:13:43.056286 1158 fs_manager.cc:359] Metadata directory not provided
I20260924 04:13:43.056350 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:13:43.056540 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:13:43.058508 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/ts-1/data/instance:
uuid: "a36ae984dedb4e8f8bed17193497bd8b"
format_stamp: "Formatted at 2026-09-24 04:13:43 on dist-test-slave-p401"
server_key: "8ba61a4ed0e61644de4e670c35a46d3c"
server_key_iv: "a8fbff450c15e4e0dcfc3d98528db241"
server_key_version: "encryptionkey@0"
I20260924 04:13:43.059026 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a36ae984dedb4e8f8bed17193497bd8b"
format_stamp: "Formatted at 2026-09-24 04:13:43 on dist-test-slave-p401"
server_key: "8ba61a4ed0e61644de4e670c35a46d3c"
server_key_iv: "a8fbff450c15e4e0dcfc3d98528db241"
server_key_version: "encryptionkey@0"
I20260924 04:13:43.062600 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:13:43.064888 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:13:43.066025 1158 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260924 04:13:43.066130 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/ts-1/wal
uuid: "a36ae984dedb4e8f8bed17193497bd8b"
format_stamp: "Formatted at 2026-09-24 04:13:43 on dist-test-slave-p401"
server_key: "8ba61a4ed0e61644de4e670c35a46d3c"
server_key_iv: "a8fbff450c15e4e0dcfc3d98528db241"
server_key_version: "encryptionkey@0"
I20260924 04:13:43.066224 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:13:43.078333 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:13:43.080983 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:13:43.081139 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:13:43.081650 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:13:43.082391 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:13:43.082433 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:13:43.082472 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:13:43.082500 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:13:43.090428 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43903
I20260924 04:13:43.090461 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43903 every 8 connection(s)
I20260924 04:13:43.091214 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:13:43.097056 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 1158
I20260924 04:13:43.097153 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:13:43.097380 392 external_mini_cluster.cc:1428] Setting key a18c3064facc3c6ef4644d261f8e4716
Sep 24 04:13:43 dist-test-slave-p401 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223223, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:13:43.104293 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:43.092897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59577 (local address 127.0.98.62:38797)
0924 04:13:43.093006 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:43.093010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:43.093766 (+ 756us) server_negotiation.cc:407] Connection header received
0924 04:13:43.095124 (+ 1358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:43.095128 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:43.095185 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:43.095269 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:43.096358 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:43.096914 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:43.098128 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:43.098339 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:43.098439 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:43.100914 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:13:43.100917 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:13:43.100919 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:13:43.100942 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:13:43.102260 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:43.102810 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:43.102813 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:43.102816 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:43.102902 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:43.103262 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:43.103265 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:43.103267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:43.103458 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:13:43.103542 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:43.104012 (+ 470us) server_negotiation.cc:299] Negotiation successful
0924 04:13:43.104133 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 04:13:43.105121 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:43.093219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38797 (local address 127.0.98.2:59577)
0924 04:13:43.093616 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:43.093675 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:43.094933 (+ 1258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:43.095443 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:43.095450 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:43.095771 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:13:43.096225 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:43.096239 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:43.097065 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:43.097068 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:43.098021 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:43.098032 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:43.098477 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:43.098481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:43.098663 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:43.099263 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:13:43.099283 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:13:43.100763 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:13:43.102372 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:43.102377 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:43.102393 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:43.102709 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:43.102991 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:43.102994 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:43.102996 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:43.103151 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:43.103554 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:13:43.103561 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:13:43.103792 (+ 231us) client_negotiation.cc:769] Sending connection context
0924 04:13:43.103957 (+ 165us) client_negotiation.cc:240] Negotiation successful
0924 04:13:43.104283 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":99,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:13:43.106244 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38797
I20260924 04:13:43.106448 1287 heartbeater.cc:464] Registering TS with master...
I20260924 04:13:43.106904 1287 heartbeater.cc:511] Master 127.0.98.62:38797 requested a full tablet report, sending...
I20260924 04:13:43.107906 962 ts_manager.cc:194] Registered new tserver with Master: a36ae984dedb4e8f8bed17193497bd8b (127.0.98.2:43903)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 04:13:43.108510 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59577
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:13:43.123991 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38797
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:13:43.232097 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:13:43.232370 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:13:43.232421 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:13:43.236411 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:13:43.236472 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:13:43.236531 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:13:43.241793 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38797
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:13:43.242851 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:13:43.243857 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:13:43.253616 1292 server_base.cc:1034] running on GCE node
W20260924 04:13:43.253542 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:13:43.253571 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:13:43.253558 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:13:43.254148 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:13:43.254801 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:13:43.255995 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790223223255968 us; error 35 us; skew 500 ppm
Sep 24 04:13:43 dist-test-slave-p401 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223223, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:13:43.258795 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:13:43.259796 1292 webserver.cc:460] Webserver started at http://127.0.98.3:39871/ using document root <none> and password file <none>
I20260924 04:13:43.260372 1292 fs_manager.cc:359] Metadata directory not provided
I20260924 04:13:43.260432 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:13:43.260602 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:13:43.262701 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/ts-2/data/instance:
uuid: "cdf389567ae04c058d12c1610a288613"
format_stamp: "Formatted at 2026-09-24 04:13:43 on dist-test-slave-p401"
server_key: "2b1872a482c7efe289cb628dd0fbc0ad"
server_key_iv: "65ad621434d9ac321fe9d8635bf4ce7a"
server_key_version: "encryptionkey@0"
I20260924 04:13:43.263262 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/ts-2/wal/instance:
uuid: "cdf389567ae04c058d12c1610a288613"
format_stamp: "Formatted at 2026-09-24 04:13:43 on dist-test-slave-p401"
server_key: "2b1872a482c7efe289cb628dd0fbc0ad"
server_key_iv: "65ad621434d9ac321fe9d8635bf4ce7a"
server_key_version: "encryptionkey@0"
I20260924 04:13:43.267030 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260924 04:13:43.269485 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:13:43.270777 1292 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:13:43.270919 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/ts-2/wal
uuid: "cdf389567ae04c058d12c1610a288613"
format_stamp: "Formatted at 2026-09-24 04:13:43 on dist-test-slave-p401"
server_key: "2b1872a482c7efe289cb628dd0fbc0ad"
server_key_iv: "65ad621434d9ac321fe9d8635bf4ce7a"
server_key_version: "encryptionkey@0"
I20260924 04:13:43.271011 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:13:43.292866 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:13:43.296239 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:13:43.296403 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:13:43.296964 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:13:43.297787 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:13:43.297844 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:13:43.297885 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:13:43.297902 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:13:43.306103 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37435
I20260924 04:13:43.306130 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37435 every 8 connection(s)
I20260924 04:13:43.307010 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:13:43.309612 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 1292
I20260924 04:13:43.309726 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790223216265354-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:13:43.310011 392 external_mini_cluster.cc:1428] Setting key 0132588ea8edc5c8a3e148a7fad1ea87
Sep 24 04:13:43 dist-test-slave-p401 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223223, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:13:43.319864 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:43.308758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59259 (local address 127.0.98.62:38797)
0924 04:13:43.308877 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:43.308880 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:43.309757 (+ 877us) server_negotiation.cc:407] Connection header received
0924 04:13:43.311376 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:43.311379 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:43.311426 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:43.311508 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:43.312792 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:43.313351 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:43.314408 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:43.314618 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:43.314687 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:43.316885 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:13:43.316888 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:13:43.316890 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:13:43.316914 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:13:43.318207 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:43.318620 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:43.318623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:43.318624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:43.318689 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:43.318940 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:43.318942 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:43.318943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:43.319102 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:13:43.319170 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:43.319624 (+ 454us) server_negotiation.cc:299] Negotiation successful
0924 04:13:43.319720 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 04:13:43.320673 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:43.309110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38797 (local address 127.0.98.3:59259)
0924 04:13:43.309592 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:43.309647 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:43.311152 (+ 1505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:43.311715 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:43.311725 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:43.312170 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:13:43.312669 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:43.312685 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:43.313442 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:43.313445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:43.314318 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:43.314326 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:43.314723 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:43.314725 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:43.314856 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:43.315401 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:13:43.315417 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:13:43.316738 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:13:43.318303 (+ 1565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:43.318308 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:43.318319 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:43.318549 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:43.318747 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:43.318749 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:43.318751 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:43.318866 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:43.319169 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:13:43.319174 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:13:43.319435 (+ 261us) client_negotiation.cc:769] Sending connection context
0924 04:13:43.319588 (+ 153us) client_negotiation.cc:240] Negotiation successful
0924 04:13:43.319839 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":113,"threads_started":1}
I20260924 04:13:43.321708 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38797
I20260924 04:13:43.321933 1421 heartbeater.cc:464] Registering TS with master...
I20260924 04:13:43.322455 1421 heartbeater.cc:511] Master 127.0.98.62:38797 requested a full tablet report, sending...
I20260924 04:13:43.323504 962 ts_manager.cc:194] Registered new tserver with Master: cdf389567ae04c058d12c1610a288613 (127.0.98.3:37435)
I20260924 04:13:43.323853 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:13:43.324112 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59259
I20260924 04:13:43.333827 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:43.327047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38472 (local address 127.0.98.62:38797)
0924 04:13:43.327165 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:43.327168 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:43.327375 (+ 207us) server_negotiation.cc:407] Connection header received
0924 04:13:43.327467 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:43.327469 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:43.327508 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:43.327577 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:43.328242 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:43.328858 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:43.329646 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:43.329862 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:43.329938 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:43.331286 (+ 1348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:13:43.331291 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:13:43.331294 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:13:43.331319 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:13:43.332459 (+ 1140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:43.332831 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:43.332834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:43.332835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:43.332915 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:43.333128 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:43.333130 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:43.333131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:43.333340 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:13:43.333451 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:43.333621 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 04:13:43.333712 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 04:13:43.338197 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38472:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:13:43.341251 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:13:43.355456 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:43.349958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43968 (local address 127.0.98.1:44969)
0924 04:13:43.350696 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:43.350708 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:43.350729 (+ 21us) server_negotiation.cc:407] Connection header received
0924 04:13:43.350803 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:43.350809 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:43.351009 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:43.351158 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:13:43.351513 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:43.352310 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:43.354172 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:43.354787 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:43.354882 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:43.355043 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:43.355167 (+ 124us) server_negotiation.cc:299] Negotiation successful
0924 04:13:43.355286 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":95,"threads_started":1}
I20260924 04:13:43.355506 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:43.349335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37435 (local address 127.0.0.1:34454)
0924 04:13:43.350296 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:43.350346 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:43.350472 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:43.351350 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:43.351353 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:43.351383 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:13:43.351579 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:43.351585 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:43.353004 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:43.353007 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:43.354017 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:43.354026 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:43.355021 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:43.355024 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:43.355110 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:43.355136 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 04:13:43.355183 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 04:13:43.355331 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":789,"mutex_wait_us":189,"spinlock_wait_cycles":8320,"thread_start_us":70,"threads_started":1}
I20260924 04:13:43.355924 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:43.349660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43903 (local address 127.0.0.1:35432)
0924 04:13:43.350665 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:43.350674 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:43.350729 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:43.351350 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:43.351353 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:43.351377 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:13:43.351562 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:43.351570 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:43.353022 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:43.353025 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:43.354265 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:43.354273 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:43.355559 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:43.355563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:43.355665 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:43.355685 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 04:13:43.355752 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 04:13:43.355810 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":956,"mutex_wait_us":109,"spinlock_wait_cycles":768,"thread_start_us":113,"threads_started":1}
I20260924 04:13:43.356040 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:43.350087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35432 (local address 127.0.98.2:43903)
0924 04:13:43.350930 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:43.350938 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:43.350954 (+ 16us) server_negotiation.cc:407] Connection header received
0924 04:13:43.351013 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:43.351017 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:43.351180 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:43.351383 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:13:43.351676 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:43.352855 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:43.354377 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:43.355322 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:43.355423 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:43.355555 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:43.355802 (+ 247us) server_negotiation.cc:299] Negotiation successful
0924 04:13:43.355897 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":762,"thread_start_us":136,"threads_started":1}
I20260924 04:13:43.355508 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:43.349460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44969 (local address 127.0.0.1:43968)
0924 04:13:43.350299 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:43.350347 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:43.350473 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:43.351167 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:43.351173 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:43.351195 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:13:43.351369 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:43.351377 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:43.352427 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:43.352431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:43.354041 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:43.354051 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:43.354895 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:43.354899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:43.354999 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:43.355034 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 04:13:43.355138 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 04:13:43.355337 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"mutex_wait_us":111,"spinlock_wait_cycles":896,"thread_start_us":102,"threads_started":1}
I20260924 04:13:43.356725 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:43.349466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34454 (local address 127.0.98.3:37435)
0924 04:13:43.350736 (+ 1270us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:43.350746 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:43.350763 (+ 17us) server_negotiation.cc:407] Connection header received
0924 04:13:43.350821 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:43.350825 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:43.351009 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:43.351158 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:13:43.351691 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:43.352863 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:43.354130 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:43.354877 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:43.356314 (+ 1437us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:43.356452 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:43.356498 (+ 46us) server_negotiation.cc:299] Negotiation successful
0924 04:13:43.356582 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1184,"thread_start_us":120,"threads_started":1}
I20260924 04:13:43.358765 1087 tablet_service.cc:1467] Processing CreateTablet for tablet a6791a8bac9b460da8cbeb44e2d4db3d (DEFAULT_TABLE table=test-table [id=2631b5ddcc9d4ab8ba15c9f9a0c32063]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:13:43.358917 1221 tablet_service.cc:1467] Processing CreateTablet for tablet a6791a8bac9b460da8cbeb44e2d4db3d (DEFAULT_TABLE table=test-table [id=2631b5ddcc9d4ab8ba15c9f9a0c32063]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:13:43.358944 1355 tablet_service.cc:1467] Processing CreateTablet for tablet a6791a8bac9b460da8cbeb44e2d4db3d (DEFAULT_TABLE table=test-table [id=2631b5ddcc9d4ab8ba15c9f9a0c32063]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:13:43.359714 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6791a8bac9b460da8cbeb44e2d4db3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:13:43.359923 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6791a8bac9b460da8cbeb44e2d4db3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:13:43.359923 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6791a8bac9b460da8cbeb44e2d4db3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:13:43.365809 1438 tablet_bootstrap.cc:492] T a6791a8bac9b460da8cbeb44e2d4db3d P 0101749bea08485a9fd7d03240f8d45a: Bootstrap starting.
I20260924 04:13:43.368125 1440 tablet_bootstrap.cc:492] T a6791a8bac9b460da8cbeb44e2d4db3d P a36ae984dedb4e8f8bed17193497bd8b: Bootstrap starting.
I20260924 04:13:43.368266 1439 tablet_bootstrap.cc:492] T a6791a8bac9b460da8cbeb44e2d4db3d P cdf389567ae04c058d12c1610a288613: Bootstrap starting.
I20260924 04:13:43.368535 1438 tablet_bootstrap.cc:654] T a6791a8bac9b460da8cbeb44e2d4db3d P 0101749bea08485a9fd7d03240f8d45a: Neither blocks nor log segments found. Creating new log.
I20260924 04:13:43.370101 1438 log.cc:824] T a6791a8bac9b460da8cbeb44e2d4db3d P 0101749bea08485a9fd7d03240f8d45a: Log is configured to *not* fsync() on all Append() calls
I20260924 04:13:43.371161 1439 tablet_bootstrap.cc:654] T a6791a8bac9b460da8cbeb44e2d4db3d P cdf389567ae04c058d12c1610a288613: Neither blocks nor log segments found. Creating new log.
I20260924 04:13:43.372123 1438 tablet_bootstrap.cc:492] T a6791a8bac9b460da8cbeb44e2d4db3d P 0101749bea08485a9fd7d03240f8d45a: No bootstrap required, opened a new log
I20260924 04:13:43.372309 1438 ts_tablet_manager.cc:1397] T a6791a8bac9b460da8cbeb44e2d4db3d P 0101749bea08485a9fd7d03240f8d45a: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 04:13:43.372551 1439 log.cc:824] T a6791a8bac9b460da8cbeb44e2d4db3d P cdf389567ae04c058d12c1610a288613: Log is configured to *not* fsync() on all Append() calls
I20260924 04:13:43.372591 1440 tablet_bootstrap.cc:654] T a6791a8bac9b460da8cbeb44e2d4db3d P a36ae984dedb4e8f8bed17193497bd8b: Neither blocks nor log segments found. Creating new log.
I20260924 04:13:43.373687 1440 log.cc:824] T a6791a8bac9b460da8cbeb44e2d4db3d P a36ae984dedb4e8f8bed17193497bd8b: Log is configured to *not* fsync() on all Append() calls
I20260924 04:13:43.374500 1439 tablet_bootstrap.cc:492] T a6791a8bac9b460da8cbeb44e2d4db3d P cdf389567ae04c058d12c1610a288613: No bootstrap required, opened a new log
I20260924 04:13:43.374681 1439 ts_tablet_manager.cc:1397] T a6791a8bac9b460da8cbeb44e2d4db3d P cdf389567ae04c058d12c1610a288613: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260924 04:13:43.375746 1440 tablet_bootstrap.cc:492] T a6791a8bac9b460da8cbeb44e2d4db3d P a36ae984dedb4e8f8bed17193497bd8b: No bootstrap required, opened a new log
I20260924 04:13:43.375916 1440 ts_tablet_manager.cc:1397] T a6791a8bac9b460da8cbeb44e2d4db3d P a36ae984dedb4e8f8bed17193497bd8b: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.004s
I20260924 04:13:43.375844 1438 raft_consensus.cc:348] T a6791a8bac9b460da8cbeb44e2d4db3d P 0101749bea08485a9fd7d03240f8d45a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf389567ae04c058d12c1610a288613" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37435 } } peers { permanent_uuid: "0101749bea08485a9fd7d03240f8d45a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44969 } } peers { permanent_uuid: "a36ae984dedb4e8f8bed17193497bd8b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } }
I20260924 04:13:43.376106 1438 raft_consensus.cc:374] T a6791a8bac9b460da8cbeb44e2d4db3d P 0101749bea08485a9fd7d03240f8d45a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:13:43.376175 1438 raft_consensus.cc:729] T a6791a8bac9b460da8cbeb44e2d4db3d P 0101749bea08485a9fd7d03240f8d45a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0101749bea08485a9fd7d03240f8d45a, State: Initialized, Role: FOLLOWER
I20260924 04:13:43.376602 1438 consensus_queue.cc:260] T a6791a8bac9b460da8cbeb44e2d4db3d P 0101749bea08485a9fd7d03240f8d45a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf389567ae04c058d12c1610a288613" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37435 } } peers { permanent_uuid: "0101749bea08485a9fd7d03240f8d45a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44969 } } peers { permanent_uuid: "a36ae984dedb4e8f8bed17193497bd8b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } }
I20260924 04:13:43.377184 1153 heartbeater.cc:503] Master 127.0.98.62:38797 was elected leader, sending a full tablet report...
I20260924 04:13:43.378700 1438 ts_tablet_manager.cc:1428] T a6791a8bac9b460da8cbeb44e2d4db3d P 0101749bea08485a9fd7d03240f8d45a: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:13:43.379513 1439 raft_consensus.cc:348] T a6791a8bac9b460da8cbeb44e2d4db3d P cdf389567ae04c058d12c1610a288613 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf389567ae04c058d12c1610a288613" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37435 } } peers { permanent_uuid: "0101749bea08485a9fd7d03240f8d45a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44969 } } peers { permanent_uuid: "a36ae984dedb4e8f8bed17193497bd8b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } }
I20260924 04:13:43.379740 1439 raft_consensus.cc:374] T a6791a8bac9b460da8cbeb44e2d4db3d P cdf389567ae04c058d12c1610a288613 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:13:43.379827 1439 raft_consensus.cc:729] T a6791a8bac9b460da8cbeb44e2d4db3d P cdf389567ae04c058d12c1610a288613 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdf389567ae04c058d12c1610a288613, State: Initialized, Role: FOLLOWER
I20260924 04:13:43.380091 1440 raft_consensus.cc:348] T a6791a8bac9b460da8cbeb44e2d4db3d P a36ae984dedb4e8f8bed17193497bd8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf389567ae04c058d12c1610a288613" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37435 } } peers { permanent_uuid: "0101749bea08485a9fd7d03240f8d45a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44969 } } peers { permanent_uuid: "a36ae984dedb4e8f8bed17193497bd8b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } }
I20260924 04:13:43.380344 1440 raft_consensus.cc:374] T a6791a8bac9b460da8cbeb44e2d4db3d P a36ae984dedb4e8f8bed17193497bd8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:13:43.380280 1439 consensus_queue.cc:260] T a6791a8bac9b460da8cbeb44e2d4db3d P cdf389567ae04c058d12c1610a288613 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf389567ae04c058d12c1610a288613" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37435 } } peers { permanent_uuid: "0101749bea08485a9fd7d03240f8d45a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44969 } } peers { permanent_uuid: "a36ae984dedb4e8f8bed17193497bd8b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } }
I20260924 04:13:43.380445 1440 raft_consensus.cc:729] T a6791a8bac9b460da8cbeb44e2d4db3d P a36ae984dedb4e8f8bed17193497bd8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a36ae984dedb4e8f8bed17193497bd8b, State: Initialized, Role: FOLLOWER
I20260924 04:13:43.380882 1421 heartbeater.cc:503] Master 127.0.98.62:38797 was elected leader, sending a full tablet report...
I20260924 04:13:43.380846 1440 consensus_queue.cc:260] T a6791a8bac9b460da8cbeb44e2d4db3d P a36ae984dedb4e8f8bed17193497bd8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf389567ae04c058d12c1610a288613" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37435 } } peers { permanent_uuid: "0101749bea08485a9fd7d03240f8d45a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44969 } } peers { permanent_uuid: "a36ae984dedb4e8f8bed17193497bd8b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } }
I20260924 04:13:43.381371 1439 ts_tablet_manager.cc:1428] T a6791a8bac9b460da8cbeb44e2d4db3d P cdf389567ae04c058d12c1610a288613: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260924 04:13:43.381758 1440 ts_tablet_manager.cc:1428] T a6791a8bac9b460da8cbeb44e2d4db3d P a36ae984dedb4e8f8bed17193497bd8b: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260924 04:13:43.382074 1287 heartbeater.cc:503] Master 127.0.98.62:38797 was elected leader, sending a full tablet report...
W20260924 04:13:43.393674 1154 tablet.cc:2369] T a6791a8bac9b460da8cbeb44e2d4db3d P 0101749bea08485a9fd7d03240f8d45a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:13:43.557936 1422 tablet.cc:2369] T a6791a8bac9b460da8cbeb44e2d4db3d P cdf389567ae04c058d12c1610a288613: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:13:43.592276 1288 tablet.cc:2369] T a6791a8bac9b460da8cbeb44e2d4db3d P a36ae984dedb4e8f8bed17193497bd8b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:13:43.783079 1446 raft_consensus.cc:482] T a6791a8bac9b460da8cbeb44e2d4db3d P a36ae984dedb4e8f8bed17193497bd8b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:13:43.783298 1446 raft_consensus.cc:504] T a6791a8bac9b460da8cbeb44e2d4db3d P a36ae984dedb4e8f8bed17193497bd8b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf389567ae04c058d12c1610a288613" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37435 } } peers { permanent_uuid: "0101749bea08485a9fd7d03240f8d45a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44969 } } peers { permanent_uuid: "a36ae984dedb4e8f8bed17193497bd8b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } }
I20260924 04:13:43.784422 1446 leader_election.cc:288] T a6791a8bac9b460da8cbeb44e2d4db3d P a36ae984dedb4e8f8bed17193497bd8b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cdf389567ae04c058d12c1610a288613 (127.0.98.3:37435), 0101749bea08485a9fd7d03240f8d45a (127.0.98.1:44969)
I20260924 04:13:43.788228 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:43.784761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38455 (local address 127.0.98.1:44969)
0924 04:13:43.784888 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:43.784892 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:43.785154 (+ 262us) server_negotiation.cc:407] Connection header received
0924 04:13:43.785254 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:43.785256 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:43.785298 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:43.785362 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:13:43.785659 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:43.786137 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:43.787291 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:43.787670 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:43.787760 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:43.787823 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:43.788024 (+ 201us) server_negotiation.cc:299] Negotiation successful
0924 04:13:43.788102 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:13:43.788233 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:43.784775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55157 (local address 127.0.98.3:37435)
0924 04:13:43.784912 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:43.784917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:43.785125 (+ 208us) server_negotiation.cc:407] Connection header received
0924 04:13:43.785246 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:43.785249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:43.785293 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:43.785353 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:13:43.785632 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:43.786137 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:43.787273 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:43.787671 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:43.787764 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:43.787826 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:43.788063 (+ 237us) server_negotiation.cc:299] Negotiation successful
0924 04:13:43.788134 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 04:13:43.788247 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:43.784665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44969 (local address 127.0.98.2:38455)
0924 04:13:43.785055 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:43.785067 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:43.785173 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:43.785397 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:43.785401 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:43.785416 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:13:43.785550 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:43.785555 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:43.786239 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:43.786242 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:43.787136 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:43.787145 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:43.787786 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:43.787790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:43.787914 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:43.787931 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 04:13:43.787989 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 04:13:43.788055 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":126,"thread_start_us":60,"threads_started":1}
I20260924 04:13:43.788252 1448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:43.784665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37435 (local address 127.0.98.2:55157)
0924 04:13:43.785013 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:43.785025 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:43.785145 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:43.785376 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:43.785379 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:43.785410 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:13:43.785532 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:43.785538 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:43.786237 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:43.786240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:43.787135 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:43.787144 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:43.787784 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:43.787788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:43.787949 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:43.787966 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 04:13:43.788038 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 04:13:43.788101 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":62,"thread_start_us":104,"threads_started":1}
I20260924 04:13:43.788937 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6791a8bac9b460da8cbeb44e2d4db3d" candidate_uuid: "a36ae984dedb4e8f8bed17193497bd8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0101749bea08485a9fd7d03240f8d45a" is_pre_election: true
I20260924 04:13:43.788938 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6791a8bac9b460da8cbeb44e2d4db3d" candidate_uuid: "a36ae984dedb4e8f8bed17193497bd8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdf389567ae04c058d12c1610a288613" is_pre_election: true
I20260924 04:13:43.789209 1107 raft_consensus.cc:2454] T a6791a8bac9b460da8cbeb44e2d4db3d P 0101749bea08485a9fd7d03240f8d45a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a36ae984dedb4e8f8bed17193497bd8b in term 0.
I20260924 04:13:43.789213 1375 raft_consensus.cc:2454] T a6791a8bac9b460da8cbeb44e2d4db3d P cdf389567ae04c058d12c1610a288613 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a36ae984dedb4e8f8bed17193497bd8b in term 0.
I20260924 04:13:43.789664 1176 leader_election.cc:302] T a6791a8bac9b460da8cbeb44e2d4db3d P a36ae984dedb4e8f8bed17193497bd8b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0101749bea08485a9fd7d03240f8d45a, a36ae984dedb4e8f8bed17193497bd8b; no voters:
I20260924 04:13:43.789914 1446 raft_consensus.cc:2781] T a6791a8bac9b460da8cbeb44e2d4db3d P a36ae984dedb4e8f8bed17193497bd8b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:13:43.789990 1446 raft_consensus.cc:482] T a6791a8bac9b460da8cbeb44e2d4db3d P a36ae984dedb4e8f8bed17193497bd8b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:13:43.790021 1446 raft_consensus.cc:3037] T a6791a8bac9b460da8cbeb44e2d4db3d P a36ae984dedb4e8f8bed17193497bd8b [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:13:43.790997 1446 raft_consensus.cc:504] T a6791a8bac9b460da8cbeb44e2d4db3d P a36ae984dedb4e8f8bed17193497bd8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf389567ae04c058d12c1610a288613" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37435 } } peers { permanent_uuid: "0101749bea08485a9fd7d03240f8d45a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44969 } } peers { permanent_uuid: "a36ae984dedb4e8f8bed17193497bd8b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } }
I20260924 04:13:43.791357 1446 leader_election.cc:288] T a6791a8bac9b460da8cbeb44e2d4db3d P a36ae984dedb4e8f8bed17193497bd8b [CANDIDATE]: Term 1 election: Requested vote from peers cdf389567ae04c058d12c1610a288613 (127.0.98.3:37435), 0101749bea08485a9fd7d03240f8d45a (127.0.98.1:44969)
I20260924 04:13:43.791647 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6791a8bac9b460da8cbeb44e2d4db3d" candidate_uuid: "a36ae984dedb4e8f8bed17193497bd8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdf389567ae04c058d12c1610a288613"
I20260924 04:13:43.791709 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6791a8bac9b460da8cbeb44e2d4db3d" candidate_uuid: "a36ae984dedb4e8f8bed17193497bd8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0101749bea08485a9fd7d03240f8d45a"
I20260924 04:13:43.791774 1375 raft_consensus.cc:3037] T a6791a8bac9b460da8cbeb44e2d4db3d P cdf389567ae04c058d12c1610a288613 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:13:43.791826 1107 raft_consensus.cc:3037] T a6791a8bac9b460da8cbeb44e2d4db3d P 0101749bea08485a9fd7d03240f8d45a [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:13:43.792954 1107 raft_consensus.cc:2454] T a6791a8bac9b460da8cbeb44e2d4db3d P 0101749bea08485a9fd7d03240f8d45a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a36ae984dedb4e8f8bed17193497bd8b in term 1.
I20260924 04:13:43.792954 1375 raft_consensus.cc:2454] T a6791a8bac9b460da8cbeb44e2d4db3d P cdf389567ae04c058d12c1610a288613 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a36ae984dedb4e8f8bed17193497bd8b in term 1.
I20260924 04:13:43.793222 1176 leader_election.cc:302] T a6791a8bac9b460da8cbeb44e2d4db3d P a36ae984dedb4e8f8bed17193497bd8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0101749bea08485a9fd7d03240f8d45a, a36ae984dedb4e8f8bed17193497bd8b, cdf389567ae04c058d12c1610a288613; no voters:
I20260924 04:13:43.793390 1446 raft_consensus.cc:2781] T a6791a8bac9b460da8cbeb44e2d4db3d P a36ae984dedb4e8f8bed17193497bd8b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:13:43.793598 1446 raft_consensus.cc:686] T a6791a8bac9b460da8cbeb44e2d4db3d P a36ae984dedb4e8f8bed17193497bd8b [term 1 LEADER]: Becoming Leader. State: Replica: a36ae984dedb4e8f8bed17193497bd8b, State: Running, Role: LEADER
I20260924 04:13:43.793938 1446 consensus_queue.cc:237] T a6791a8bac9b460da8cbeb44e2d4db3d P a36ae984dedb4e8f8bed17193497bd8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf389567ae04c058d12c1610a288613" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37435 } } peers { permanent_uuid: "0101749bea08485a9fd7d03240f8d45a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44969 } } peers { permanent_uuid: "a36ae984dedb4e8f8bed17193497bd8b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } }
I20260924 04:13:43.796990 961 catalog_manager.cc:5495] T a6791a8bac9b460da8cbeb44e2d4db3d P a36ae984dedb4e8f8bed17193497bd8b reported cstate change: term changed from 0 to 1, leader changed from <none> to a36ae984dedb4e8f8bed17193497bd8b (127.0.98.2). New cstate: current_term: 1 leader_uuid: "a36ae984dedb4e8f8bed17193497bd8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf389567ae04c058d12c1610a288613" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0101749bea08485a9fd7d03240f8d45a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a36ae984dedb4e8f8bed17193497bd8b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } health_report { overall_health: HEALTHY } } }
Sep 24 04:13:43 dist-test-slave-p401 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:13:43.922299 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:43.908121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54640 (local address 127.0.98.1:44969)
0924 04:13:43.908265 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:43.908269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:43.908365 (+ 96us) server_negotiation.cc:407] Connection header received
0924 04:13:43.908488 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:43.908491 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:43.908534 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:43.908599 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:43.909226 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:43.909728 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:43.910520 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:43.910731 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:43.910828 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:43.919448 (+ 8620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:13:43.919460 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:13:43.919475 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:13:43.919502 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:13:43.920903 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:43.921281 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:43.921286 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:43.921298 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:43.921405 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:43.921587 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:43.921589 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:43.921590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:43.921817 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:13:43.921959 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:43.922082 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 04:13:43.922174 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Sep 24 04:13:43 dist-test-slave-p401 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223223, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 04:13:43 dist-test-slave-p401 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223223, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:13:43.946362 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:43.938772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54654 (local address 127.0.98.1:44969)
0924 04:13:43.938909 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:43.938913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:43.939044 (+ 131us) server_negotiation.cc:407] Connection header received
0924 04:13:43.939168 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:43.939170 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:43.939212 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:43.939306 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:43.939939 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:43.940442 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:43.941230 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:43.941452 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:43.941538 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:43.943650 (+ 2112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:13:43.943653 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:13:43.943655 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:13:43.943678 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:13:43.944902 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:43.945256 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:43.945260 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:43.945262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:43.945332 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:43.945560 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:43.945563 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:43.945565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:43.945723 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:13:43.945849 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:43.946103 (+ 254us) server_negotiation.cc:299] Negotiation successful
0924 04:13:43.946214 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Sep 24 04:13:43 dist-test-slave-p401 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 04:13:43 dist-test-slave-p401 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:13:43.969901 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:43.962406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54662 (local address 127.0.98.1:44969)
0924 04:13:43.962539 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:43.962543 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:43.962669 (+ 126us) server_negotiation.cc:407] Connection header received
0924 04:13:43.962798 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:43.962801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:43.962842 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:43.962915 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:43.963553 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:43.964040 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:43.964824 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:43.965026 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:43.965089 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:43.967308 (+ 2219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:13:43.967312 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:13:43.967315 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:13:43.967337 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:13:43.968405 (+ 1068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:43.968809 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:43.968811 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:43.968812 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:43.968882 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:43.969124 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:43.969128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:43.969129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:43.969284 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:13:43.969397 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:43.969620 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 04:13:43.969748 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260924 04:13:43.971138 1067 tablet_service.cc:2908] Rejecting scan request for tablet a6791a8bac9b460da8cbeb44e2d4db3d: Uninitialized: clean time has not yet been initialized
I20260924 04:13:43.972906 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 1024
I20260924 04:13:43.978474 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 1158
I20260924 04:13:43.982820 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 1292
I20260924 04:13:43.986944 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 930
2026-09-24T04:13:43Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2608 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:13:44 dist-test-slave-p401 krb5kdc[1463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:13:44 dist-test-slave-p401 krb5kdc[1463](info): set up 2 sockets
Sep 24 04:13:44 dist-test-slave-p401 krb5kdc[1463](info): commencing operation
krb5kdc: starting...
W20260924 04:13:44.946667 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/krb5kdc/test-user.keytab.
Sep 24 04:13:45 dist-test-slave-p401 krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:13:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:13:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:13:45.057991 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36995
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36995
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:13:45.163442 1476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:13:45.163722 1476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:13:45.163787 1476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:13:45.167774 1476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:13:45.167838 1476 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 04:13:45.167865 1476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:13:45.167894 1476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:13:45.167935 1476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:13:45.167963 1476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:13:45.173429 1476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36995
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:13:45.174439 1476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:13:45.175483 1476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:13:45.183683 1482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:13:45.183676 1481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:13:45.183809 1476 server_base.cc:1034] running on GCE node
W20260924 04:13:45.183677 1484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:13:45.184527 1476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:13:45.185539 1476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:13:45.186683 1476 hybrid_clock.cc:648] HybridClock initialized: now 1790223225186658 us; error 41 us; skew 500 ppm
Sep 24 04:13:45 dist-test-slave-p401 krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:13:45.189327 1476 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:13:45.190160 1476 webserver.cc:460] Webserver started at http://127.0.98.62:45533/ using document root <none> and password file <none>
I20260924 04:13:45.190680 1476 fs_manager.cc:359] Metadata directory not provided
I20260924 04:13:45.190742 1476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:13:45.190924 1476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:13:45.192998 1476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/master-0/data/instance:
uuid: "8e587b56a18342baa915c856431b902a"
format_stamp: "Formatted at 2026-09-24 04:13:45 on dist-test-slave-p401"
server_key: "30eacaae7ef98238c69d591f988b2b9f"
server_key_iv: "9d0732514e3fddf80efebab15066bfbb"
server_key_version: "encryptionkey@0"
I20260924 04:13:45.193534 1476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/master-0/wal/instance:
uuid: "8e587b56a18342baa915c856431b902a"
format_stamp: "Formatted at 2026-09-24 04:13:45 on dist-test-slave-p401"
server_key: "30eacaae7ef98238c69d591f988b2b9f"
server_key_iv: "9d0732514e3fddf80efebab15066bfbb"
server_key_version: "encryptionkey@0"
I20260924 04:13:45.196969 1476 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:13:45.199178 1491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:13:45.200356 1476 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 04:13:45.200477 1476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/master-0/wal
uuid: "8e587b56a18342baa915c856431b902a"
format_stamp: "Formatted at 2026-09-24 04:13:45 on dist-test-slave-p401"
server_key: "30eacaae7ef98238c69d591f988b2b9f"
server_key_iv: "9d0732514e3fddf80efebab15066bfbb"
server_key_version: "encryptionkey@0"
I20260924 04:13:45.200562 1476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:13:45.209817 1476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:13:45.212785 1476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:13:45.212949 1476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:13:45.219847 1476 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36995
I20260924 04:13:45.219858 1553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36995 every 8 connection(s)
I20260924 04:13:45.220726 1476 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:13:45.222872 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 1476
I20260924 04:13:45.222961 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/master-0/wal/instance
I20260924 04:13:45.223186 392 external_mini_cluster.cc:1428] Setting key 1ac0e08454d3a812ecb77335b2a101b5
I20260924 04:13:45.223351 1554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:13:45.229882 1554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e587b56a18342baa915c856431b902a: Bootstrap starting.
Sep 24 04:13:45 dist-test-slave-p401 krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:13:45.232193 1554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e587b56a18342baa915c856431b902a: Neither blocks nor log segments found. Creating new log.
I20260924 04:13:45.232990 1554 log.cc:824] T 00000000000000000000000000000000 P 8e587b56a18342baa915c856431b902a: Log is configured to *not* fsync() on all Append() calls
I20260924 04:13:45.234895 1554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e587b56a18342baa915c856431b902a: No bootstrap required, opened a new log
I20260924 04:13:45.237039 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:45.224489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34808 (local address 127.0.98.62:36995)
0924 04:13:45.224878 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:45.224905 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:45.224946 (+ 41us) server_negotiation.cc:407] Connection header received
0924 04:13:45.225825 (+ 879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:45.225848 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:45.226251 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:45.226601 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:45.227201 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:45.228629 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:45.229420 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:45.230070 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:45.230215 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:45.232481 (+ 2266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:13:45.232487 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:13:45.232502 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:13:45.232531 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:13:45.234490 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:45.234934 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:45.234938 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:45.234942 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:45.235065 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:45.235354 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:45.235356 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:45.235357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:45.235707 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:13:45.235848 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:45.236105 (+ 257us) server_negotiation.cc:299] Negotiation successful
0924 04:13:45.236354 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":105,"threads_started":1}
I20260924 04:13:45.238000 1554 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e587b56a18342baa915c856431b902a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e587b56a18342baa915c856431b902a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36995 } }
I20260924 04:13:45.238207 1554 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e587b56a18342baa915c856431b902a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:13:45.238261 1554 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e587b56a18342baa915c856431b902a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e587b56a18342baa915c856431b902a, State: Initialized, Role: FOLLOWER
I20260924 04:13:45.238665 1554 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e587b56a18342baa915c856431b902a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e587b56a18342baa915c856431b902a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36995 } }
I20260924 04:13:45.238798 1554 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e587b56a18342baa915c856431b902a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:13:45.238893 1554 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e587b56a18342baa915c856431b902a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:13:45.238996 1554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e587b56a18342baa915c856431b902a [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:13:45.239863 1554 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e587b56a18342baa915c856431b902a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e587b56a18342baa915c856431b902a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36995 } }
I20260924 04:13:45.240183 1554 leader_election.cc:302] T 00000000000000000000000000000000 P 8e587b56a18342baa915c856431b902a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e587b56a18342baa915c856431b902a; no voters:
I20260924 04:13:45.240460 1554 leader_election.cc:288] T 00000000000000000000000000000000 P 8e587b56a18342baa915c856431b902a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:13:45.240568 1559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e587b56a18342baa915c856431b902a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:13:45.240758 1559 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e587b56a18342baa915c856431b902a [term 1 LEADER]: Becoming Leader. State: Replica: 8e587b56a18342baa915c856431b902a, State: Running, Role: LEADER
I20260924 04:13:45.241079 1559 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e587b56a18342baa915c856431b902a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e587b56a18342baa915c856431b902a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36995 } }
I20260924 04:13:45.241575 1554 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e587b56a18342baa915c856431b902a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:13:45.242657 1561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e587b56a18342baa915c856431b902a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e587b56a18342baa915c856431b902a. Latest consensus state: current_term: 1 leader_uuid: "8e587b56a18342baa915c856431b902a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e587b56a18342baa915c856431b902a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36995 } } }
I20260924 04:13:45.242779 1561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e587b56a18342baa915c856431b902a [sys.catalog]: This master's current role is: LEADER
I20260924 04:13:45.243029 1560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e587b56a18342baa915c856431b902a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e587b56a18342baa915c856431b902a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e587b56a18342baa915c856431b902a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36995 } } }
I20260924 04:13:45.243108 1560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e587b56a18342baa915c856431b902a [sys.catalog]: This master's current role is: LEADER
I20260924 04:13:45.243132 1568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:13:45.245625 1568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:13:45.250905 1568 catalog_manager.cc:1347] Generated new cluster ID: 33f13fb3ace14a38bfadb07faef315c6
I20260924 04:13:45.250967 1568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:13:45.272537 1568 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:13:45.273787 1568 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:13:45.283944 1568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e587b56a18342baa915c856431b902a: Generated new TSK 0
I20260924 04:13:45.284791 1568 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:13:45.310252 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36995
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:13:45.416829 1580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:13:45.417104 1580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:13:45.417167 1580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:13:45.421201 1580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:13:45.421260 1580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:13:45.421316 1580 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 04:13:45.421371 1580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:13:45.426451 1580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:13:45.427476 1580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:13:45.428439 1580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:13:45.437462 1586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:13:45.437548 1580 server_base.cc:1034] running on GCE node
W20260924 04:13:45.437486 1588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:13:45.437486 1585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:13:45.438081 1580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:13:45.438665 1580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:13:45.439834 1580 hybrid_clock.cc:648] HybridClock initialized: now 1790223225439811 us; error 33 us; skew 500 ppm
Sep 24 04:13:45 dist-test-slave-p401 krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:13:45.442420 1580 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:13:45.443387 1580 webserver.cc:460] Webserver started at http://127.0.98.1:34247/ using document root <none> and password file <none>
I20260924 04:13:45.443909 1580 fs_manager.cc:359] Metadata directory not provided
I20260924 04:13:45.443972 1580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:13:45.444149 1580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:13:45.446095 1580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/ts-0/data/instance:
uuid: "d0a557b2cdf74293894448646cb25674"
format_stamp: "Formatted at 2026-09-24 04:13:45 on dist-test-slave-p401"
server_key: "a197f6580eea3204ab5d84d213f5607e"
server_key_iv: "830275cd4f3483d9c4328ab5bf654d8c"
server_key_version: "encryptionkey@0"
I20260924 04:13:45.446614 1580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d0a557b2cdf74293894448646cb25674"
format_stamp: "Formatted at 2026-09-24 04:13:45 on dist-test-slave-p401"
server_key: "a197f6580eea3204ab5d84d213f5607e"
server_key_iv: "830275cd4f3483d9c4328ab5bf654d8c"
server_key_version: "encryptionkey@0"
I20260924 04:13:45.450055 1580 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260924 04:13:45.452382 1595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:13:45.453562 1580 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:13:45.453672 1580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/ts-0/wal
uuid: "d0a557b2cdf74293894448646cb25674"
format_stamp: "Formatted at 2026-09-24 04:13:45 on dist-test-slave-p401"
server_key: "a197f6580eea3204ab5d84d213f5607e"
server_key_iv: "830275cd4f3483d9c4328ab5bf654d8c"
server_key_version: "encryptionkey@0"
I20260924 04:13:45.453758 1580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:13:45.464435 1580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:13:45.467002 1580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:13:45.467164 1580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:13:45.468492 1580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:13:45.468539 1580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:13:45.468590 1580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:13:45.468619 1580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 04:13:45 dist-test-slave-p401 krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:13:45.478609 1580 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42723
I20260924 04:13:45.478749 1710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42723 every 8 connection(s)
I20260924 04:13:45.479494 1580 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:13:45.481541 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:45.469253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41499 (local address 127.0.98.62:36995)
0924 04:13:45.469354 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:45.469358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:45.470353 (+ 995us) server_negotiation.cc:407] Connection header received
0924 04:13:45.471372 (+ 1019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:45.471376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:45.471462 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:45.471522 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:45.473382 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:45.473974 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:45.475166 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:45.475457 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:45.475552 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:45.478053 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:13:45.478057 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:13:45.478060 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:13:45.478087 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:13:45.479533 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:45.480104 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:45.480107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:45.480109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:45.480198 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:45.480499 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:45.480502 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:45.480504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:45.480699 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:13:45.480777 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:45.481249 (+ 472us) server_negotiation.cc:299] Negotiation successful
0924 04:13:45.481368 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 04:13:45.482260 1604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:45.469780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.1:41499)
0924 04:13:45.470220 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:45.470259 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:45.471111 (+ 852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:45.471871 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:45.471877 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:45.472220 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:13:45.473123 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:45.473149 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:45.474088 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:45.474091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:45.475060 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:45.475069 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:45.475629 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:45.475634 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:45.475816 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:45.476349 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:13:45.476363 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:13:45.477898 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:13:45.479634 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:45.479640 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:45.479654 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:45.479983 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:45.480281 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:45.480284 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:45.480286 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:45.480414 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:45.480797 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:13:45.480804 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:13:45.481034 (+ 230us) client_negotiation.cc:769] Sending connection context
0924 04:13:45.481193 (+ 159us) client_negotiation.cc:240] Negotiation successful
0924 04:13:45.481487 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":126,"threads_started":1}
I20260924 04:13:45.483449 1711 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36995
I20260924 04:13:45.483588 1711 heartbeater.cc:464] Registering TS with master...
I20260924 04:13:45.484143 1711 heartbeater.cc:511] Master 127.0.98.62:36995 requested a full tablet report, sending...
I20260924 04:13:45.485652 1507 ts_manager.cc:194] Registered new tserver with Master: d0a557b2cdf74293894448646cb25674 (127.0.98.1:42723)
I20260924 04:13:45.485914 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 1580
I20260924 04:13:45.486017 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:13:45.486299 392 external_mini_cluster.cc:1428] Setting key 8bbddc7224c0182e8177aef839df4a54
I20260924 04:13:45.487000 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41499
I20260924 04:13:45.492434 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:45.485559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41641 (local address 127.0.98.62:36995)
0924 04:13:45.485666 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:45.485670 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:45.485802 (+ 132us) server_negotiation.cc:407] Connection header received
0924 04:13:45.485973 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:45.485976 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:45.486020 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:45.486093 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:45.486627 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:45.487356 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:45.488401 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:45.488704 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:45.488809 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:45.489833 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:13:45.489835 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:13:45.489837 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:13:45.489856 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:13:45.491011 (+ 1155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:45.491416 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:45.491420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:45.491422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:45.491504 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:45.491730 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:45.491733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:45.491735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:45.491897 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:13:45.491976 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:45.492198 (+ 222us) server_negotiation.cc:299] Negotiation successful
0924 04:13:45.492295 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 04:13:45.492453 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:45.485468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.1:41641)
0924 04:13:45.485725 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:45.485740 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:45.485862 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:45.486196 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:45.486198 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:45.486384 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:13:45.486518 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:45.486526 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:45.487474 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:45.487477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:45.488273 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:45.488284 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:45.488822 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:45.488825 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:45.488914 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:45.489316 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:13:45.489329 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:13:45.489714 (+ 385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:13:45.491104 (+ 1390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:45.491107 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:45.491109 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:45.491337 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:45.491579 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:45.491582 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:45.491584 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:45.491658 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:45.491977 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:13:45.491980 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:13:45.492060 (+ 80us) client_negotiation.cc:769] Sending connection context
0924 04:13:45.492141 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 04:13:45.492262 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":89,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:13:45.510355 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36995
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:13:45.615423 1720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:13:45.615665 1720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:13:45.615705 1720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:13:45.619572 1720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:13:45.619624 1720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:13:45.619661 1720 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 04:13:45.619697 1720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:13:45.624665 1720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:13:45.625625 1720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:13:45.626537 1720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:13:45.635414 1725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:13:45.635352 1728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:13:45.635347 1726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:13:45.636008 1720 server_base.cc:1034] running on GCE node
I20260924 04:13:45.636360 1720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:13:45.636922 1720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:13:45.638094 1720 hybrid_clock.cc:648] HybridClock initialized: now 1790223225638075 us; error 29 us; skew 500 ppm
Sep 24 04:13:45 dist-test-slave-p401 krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:13:45.644680 1720 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:13:45.645515 1720 webserver.cc:460] Webserver started at http://127.0.98.2:35191/ using document root <none> and password file <none>
I20260924 04:13:45.646019 1720 fs_manager.cc:359] Metadata directory not provided
I20260924 04:13:45.646081 1720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:13:45.646253 1720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:13:45.648203 1720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/ts-1/data/instance:
uuid: "a552ee34a332411b8c448115dc3e369d"
format_stamp: "Formatted at 2026-09-24 04:13:45 on dist-test-slave-p401"
server_key: "4017708f74641a71035ee17adfc19b6f"
server_key_iv: "437ec267134e04bfe842a44bb9f0dd01"
server_key_version: "encryptionkey@0"
I20260924 04:13:45.648701 1720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a552ee34a332411b8c448115dc3e369d"
format_stamp: "Formatted at 2026-09-24 04:13:45 on dist-test-slave-p401"
server_key: "4017708f74641a71035ee17adfc19b6f"
server_key_iv: "437ec267134e04bfe842a44bb9f0dd01"
server_key_version: "encryptionkey@0"
I20260924 04:13:45.652114 1720 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:13:45.654487 1735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:13:45.655659 1720 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:13:45.655757 1720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/ts-1/wal
uuid: "a552ee34a332411b8c448115dc3e369d"
format_stamp: "Formatted at 2026-09-24 04:13:45 on dist-test-slave-p401"
server_key: "4017708f74641a71035ee17adfc19b6f"
server_key_iv: "437ec267134e04bfe842a44bb9f0dd01"
server_key_version: "encryptionkey@0"
I20260924 04:13:45.655840 1720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:13:45.671798 1720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:13:45.674645 1720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:13:45.674804 1720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:13:45.676170 1720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:13:45.676216 1720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:13:45.676265 1720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:13:45.676294 1720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 04:13:45 dist-test-slave-p401 krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:13:45.685958 1720 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44665
I20260924 04:13:45.686041 1850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44665 every 8 connection(s)
I20260924 04:13:45.686826 1720 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:13:45.688508 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:45.676917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57273 (local address 127.0.98.62:36995)
0924 04:13:45.677024 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:45.677027 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:45.678005 (+ 978us) server_negotiation.cc:407] Connection header received
0924 04:13:45.678968 (+ 963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:45.678972 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:45.679053 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:45.679143 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:45.680453 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:45.680966 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:45.682045 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:45.682319 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:45.682421 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:45.685082 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:13:45.685085 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:13:45.685088 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:13:45.685115 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:13:45.686496 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:45.687009 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:45.687012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:45.687013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:45.687088 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:45.687474 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:45.687477 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:45.687479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:45.687678 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:13:45.687762 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:45.688227 (+ 465us) server_negotiation.cc:299] Negotiation successful
0924 04:13:45.688344 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 04:13:45.689188 1744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:45.677430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.2:57273)
0924 04:13:45.677859 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:45.677910 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:45.678732 (+ 822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:45.679365 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:45.679375 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:45.679754 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:13:45.680308 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:45.680323 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:45.681078 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:45.681081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:45.681924 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:45.681933 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:45.682464 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:45.682468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:45.682630 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:45.683141 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:13:45.683161 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:13:45.684901 (+ 1740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:13:45.686597 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:45.686602 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:45.686615 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:45.686920 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:45.687168 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:45.687171 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:45.687173 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:45.687350 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:45.687779 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:13:45.687785 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:13:45.688010 (+ 225us) client_negotiation.cc:769] Sending connection context
0924 04:13:45.688154 (+ 144us) client_negotiation.cc:240] Negotiation successful
0924 04:13:45.688440 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":124,"threads_started":1}
I20260924 04:13:45.690222 1851 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36995
I20260924 04:13:45.690344 1851 heartbeater.cc:464] Registering TS with master...
I20260924 04:13:45.690794 1851 heartbeater.cc:511] Master 127.0.98.62:36995 requested a full tablet report, sending...
I20260924 04:13:45.691735 1508 ts_manager.cc:194] Registered new tserver with Master: a552ee34a332411b8c448115dc3e369d (127.0.98.2:44665)
I20260924 04:13:45.692404 1508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57273
I20260924 04:13:45.695884 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 1720
I20260924 04:13:45.696012 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:13:45.696352 392 external_mini_cluster.cc:1428] Setting key 6a3d5aa55e4e305b2974cb50f5ebb145
I20260924 04:13:45.698486 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:45.692143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45905 (local address 127.0.98.62:36995)
0924 04:13:45.692243 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:45.692247 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:45.692392 (+ 145us) server_negotiation.cc:407] Connection header received
0924 04:13:45.692493 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:45.692496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:45.692535 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:45.692595 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:45.692975 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:45.693452 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:45.694210 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:45.694409 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:45.694488 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:45.695559 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:13:45.695564 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:13:45.695567 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:13:45.695595 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:13:45.696976 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:45.697343 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:45.697346 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:45.697348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:45.697436 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:45.697747 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:45.697750 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:45.697751 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:45.697955 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:13:45.698029 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:45.698298 (+ 269us) server_negotiation.cc:299] Negotiation successful
0924 04:13:45.698382 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 04:13:45.698674 1857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:45.692088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.2:45905)
0924 04:13:45.692324 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:45.692335 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:45.692425 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:45.692617 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:45.692619 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:45.692791 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:13:45.692876 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:45.692883 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:45.693559 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:45.693562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:45.694112 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:45.694118 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:45.694500 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:45.694503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:45.694586 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:45.695042 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:13:45.695054 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:13:45.695427 (+ 373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:13:45.697076 (+ 1649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:45.697078 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:45.697079 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:45.697257 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:45.697542 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:45.697546 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:45.697548 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:45.697643 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:45.698047 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:13:45.698050 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:13:45.698152 (+ 102us) client_negotiation.cc:769] Sending connection context
0924 04:13:45.698266 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 04:13:45.698404 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":92,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:13:45.721695 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36995
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:13:45.827564 1860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:13:45.827821 1860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:13:45.827879 1860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:13:45.831923 1860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:13:45.831987 1860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:13:45.832037 1860 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 04:13:45.832085 1860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:13:45.837164 1860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:13:45.838069 1860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:13:45.838976 1860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:13:45.847874 1865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:13:45.847892 1866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:13:45.847954 1868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:13:45.848044 1860 server_base.cc:1034] running on GCE node
I20260924 04:13:45.848567 1860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:13:45.849146 1860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:13:45.850294 1860 hybrid_clock.cc:648] HybridClock initialized: now 1790223225850277 us; error 32 us; skew 500 ppm
Sep 24 04:13:45 dist-test-slave-p401 krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:13:45.852756 1860 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:13:45.853525 1860 webserver.cc:460] Webserver started at http://127.0.98.3:37175/ using document root <none> and password file <none>
I20260924 04:13:45.854022 1860 fs_manager.cc:359] Metadata directory not provided
I20260924 04:13:45.854082 1860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:13:45.854264 1860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:13:45.856261 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/ts-2/data/instance:
uuid: "3f8b5a5b325842fa82175c27f3eec913"
format_stamp: "Formatted at 2026-09-24 04:13:45 on dist-test-slave-p401"
server_key: "99f0a2d4602b207362be84a6d3cb2e90"
server_key_iv: "37a1d1816e9b05045b2921a7fc1b249e"
server_key_version: "encryptionkey@0"
I20260924 04:13:45.856827 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3f8b5a5b325842fa82175c27f3eec913"
format_stamp: "Formatted at 2026-09-24 04:13:45 on dist-test-slave-p401"
server_key: "99f0a2d4602b207362be84a6d3cb2e90"
server_key_iv: "37a1d1816e9b05045b2921a7fc1b249e"
server_key_version: "encryptionkey@0"
I20260924 04:13:45.860366 1860 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:13:45.862571 1875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:13:45.863721 1860 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:13:45.863819 1860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/ts-2/wal
uuid: "3f8b5a5b325842fa82175c27f3eec913"
format_stamp: "Formatted at 2026-09-24 04:13:45 on dist-test-slave-p401"
server_key: "99f0a2d4602b207362be84a6d3cb2e90"
server_key_iv: "37a1d1816e9b05045b2921a7fc1b249e"
server_key_version: "encryptionkey@0"
I20260924 04:13:45.863904 1860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:13:45.879807 1860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:13:45.882347 1860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:13:45.882513 1860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:13:45.883929 1860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:13:45.883978 1860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:13:45.884029 1860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:13:45.884055 1860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 04:13:45 dist-test-slave-p401 krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:13:45.893451 1860 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35997
I20260924 04:13:45.893565 1990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35997 every 8 connection(s)
I20260924 04:13:45.894310 1860 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:13:45.895732 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:45.884630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59137 (local address 127.0.98.62:36995)
0924 04:13:45.884771 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:45.884777 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:45.885573 (+ 796us) server_negotiation.cc:407] Connection header received
0924 04:13:45.886635 (+ 1062us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:45.886639 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:45.886699 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:45.886797 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:45.888022 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:45.888545 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:45.889637 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:45.889896 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:45.889980 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:45.892423 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:13:45.892427 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:13:45.892430 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:13:45.892456 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:13:45.893841 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:45.894284 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:45.894288 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:45.894290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:45.894369 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:45.894678 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:45.894681 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:45.894683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:45.894863 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:13:45.894960 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:45.895517 (+ 557us) server_negotiation.cc:299] Negotiation successful
0924 04:13:45.895617 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 04:13:45.896497 1884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:45.884991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.3:59137)
0924 04:13:45.885421 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:45.885469 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:45.886417 (+ 948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:45.886985 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:45.886994 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:45.887394 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:13:45.887879 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:45.887901 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:45.888653 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:45.888656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:45.889531 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:45.889539 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:45.890046 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:45.890049 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:45.890216 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:45.890762 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:13:45.890780 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:13:45.892265 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:13:45.893938 (+ 1673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:45.893944 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:45.893959 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:45.894195 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:45.894450 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:45.894453 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:45.894455 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:45.894594 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:45.895010 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:13:45.895016 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:13:45.895307 (+ 291us) client_negotiation.cc:769] Sending connection context
0924 04:13:45.895456 (+ 149us) client_negotiation.cc:240] Negotiation successful
0924 04:13:45.895690 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":99,"threads_started":1}
I20260924 04:13:45.896822 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 1860
I20260924 04:13:45.896924 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:13:45.897202 392 external_mini_cluster.cc:1428] Setting key b3da88fe4a010a594894ae8cf9e104ba
I20260924 04:13:45.897871 1991 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36995
I20260924 04:13:45.898010 1991 heartbeater.cc:464] Registering TS with master...
I20260924 04:13:45.898541 1991 heartbeater.cc:511] Master 127.0.98.62:36995 requested a full tablet report, sending...
I20260924 04:13:45.899501 1507 ts_manager.cc:194] Registered new tserver with Master: 3f8b5a5b325842fa82175c27f3eec913 (127.0.98.3:35997)
I20260924 04:13:45.900094 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59137
I20260924 04:13:45.906183 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:45.900015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43485 (local address 127.0.98.62:36995)
0924 04:13:45.900112 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:45.900116 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:45.900340 (+ 224us) server_negotiation.cc:407] Connection header received
0924 04:13:45.900438 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:45.900440 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:45.900481 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:45.900537 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:45.900948 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:45.901668 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:45.902415 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:45.902617 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:45.902689 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:45.903717 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:13:45.903720 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:13:45.903722 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:13:45.903743 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:13:45.905004 (+ 1261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:45.905334 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:45.905336 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:45.905337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:45.905422 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:45.905604 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:45.905605 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:45.905606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:45.905749 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:13:45.905813 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:45.905997 (+ 184us) server_negotiation.cc:299] Negotiation successful
0924 04:13:45.906078 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 04:13:45.906191 1997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:45.899915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.3:43485)
0924 04:13:45.900255 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:45.900269 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:45.900363 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:45.900569 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:45.900572 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:45.900749 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:13:45.900841 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:45.900846 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:45.901770 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:45.901772 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:45.902319 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:45.902326 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:45.902711 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:45.902714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:45.902783 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:45.903202 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:13:45.903212 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:13:45.903609 (+ 397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:13:45.905089 (+ 1480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:45.905092 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:45.905094 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:45.905267 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:45.905481 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:45.905482 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:45.905484 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:45.905547 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:45.905808 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:13:45.905810 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:13:45.905887 (+ 77us) client_negotiation.cc:769] Sending connection context
0924 04:13:45.905956 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 04:13:45.906053 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":97,"threads_started":1}
I20260924 04:13:45.910488 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 04:13:45 dist-test-slave-p401 krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223225, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 04:13:45 dist-test-slave-p401 krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223225, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:13:45.932336 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:45.925501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34822 (local address 127.0.98.62:36995)
0924 04:13:45.925614 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:45.925618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:45.925629 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:13:45.925666 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:45.925668 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:45.925705 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:45.925772 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:45.926347 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:45.926841 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:45.927661 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:45.927865 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:45.927946 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:45.929985 (+ 2039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:13:45.929988 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:13:45.929990 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:13:45.930012 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:13:45.931059 (+ 1047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:45.931448 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:45.931451 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:45.931452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:45.931523 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:45.931732 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:45.931734 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:45.931735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:45.931896 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:13:45.931986 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:45.932140 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 04:13:45.932222 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 04:13:45.934914 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34822:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:13:45.937764 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:13:45.950806 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:45.945335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42723 (local address 127.0.0.1:38730)
0924 04:13:45.945783 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:45.945834 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:45.945957 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:45.947306 (+ 1349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:45.947310 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:45.947327 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:13:45.947430 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:45.947436 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:45.948479 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:45.948482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:45.949322 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:45.949332 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:45.950266 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:45.950270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:45.950370 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:45.950420 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 04:13:45.950518 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 04:13:45.950634 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"spinlock_wait_cycles":2304,"thread_start_us":112,"threads_started":1}
I20260924 04:13:45.950793 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:45.946301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38730 (local address 127.0.98.1:42723)
0924 04:13:45.946836 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:45.946847 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:45.946869 (+ 22us) server_negotiation.cc:407] Connection header received
0924 04:13:45.946939 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:45.946943 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:45.947126 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:45.947316 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:13:45.947538 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:45.948359 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:45.949453 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:45.950159 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:45.950252 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:45.950414 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:45.950567 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 04:13:45.950652 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":89,"threads_started":1}
I20260924 04:13:45.950840 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:45.945253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44665 (local address 127.0.0.1:55828)
0924 04:13:45.945783 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:45.945834 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:45.945957 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:45.946446 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:45.946451 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:45.946474 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:13:45.946596 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:45.946603 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:45.948080 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:45.948084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:45.949408 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:45.949417 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:45.950455 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:45.950458 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:45.950542 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:45.950568 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 04:13:45.950613 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 04:13:45.950678 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"mutex_wait_us":1,"spinlock_wait_cycles":7808,"thread_start_us":64,"threads_started":1}
I20260924 04:13:45.951150 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:45.945925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35997 (local address 127.0.0.1:56406)
0924 04:13:45.946741 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:45.946753 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:45.946831 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:45.947413 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:45.947415 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:45.947431 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:13:45.947516 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:45.947523 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:45.948593 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:45.948596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:45.949677 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:45.949684 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:45.950825 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:45.950829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:45.950904 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:45.950924 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 04:13:45.950987 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 04:13:45.951045 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":763,"thread_start_us":86,"threads_started":1}
I20260924 04:13:45.951467 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:45.945397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55828 (local address 127.0.98.2:44665)
0924 04:13:45.945976 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:45.945988 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:45.946013 (+ 25us) server_negotiation.cc:407] Connection header received
0924 04:13:45.946082 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:45.946087 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:45.946280 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:45.946438 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:13:45.946808 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:45.947970 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:45.949536 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:45.950322 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:45.951003 (+ 681us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:45.951147 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:45.951196 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 04:13:45.951321 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":96,"threads_started":1}
I20260924 04:13:45.951709 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:45.946286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56406 (local address 127.0.98.3:35997)
0924 04:13:45.946671 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:45.946681 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:45.946921 (+ 240us) server_negotiation.cc:407] Connection header received
0924 04:13:45.946984 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:45.946988 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:45.947126 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:45.947284 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:13:45.947610 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:45.948489 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:45.949792 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:45.950544 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:45.950630 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:45.950773 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:45.951498 (+ 725us) server_negotiation.cc:299] Negotiation successful
0924 04:13:45.951585 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":109,"threads_started":1}
I20260924 04:13:45.953958 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 16c9dce032904892a39472a742b13d3e (DEFAULT_TABLE table=test-table [id=5f95375d40a24acf9999ac9bdf58c41f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:13:45.953980 1645 tablet_service.cc:1467] Processing CreateTablet for tablet 16c9dce032904892a39472a742b13d3e (DEFAULT_TABLE table=test-table [id=5f95375d40a24acf9999ac9bdf58c41f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:13:45.953981 1785 tablet_service.cc:1467] Processing CreateTablet for tablet 16c9dce032904892a39472a742b13d3e (DEFAULT_TABLE table=test-table [id=5f95375d40a24acf9999ac9bdf58c41f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:13:45.954931 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16c9dce032904892a39472a742b13d3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:13:45.954943 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16c9dce032904892a39472a742b13d3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:13:45.954943 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16c9dce032904892a39472a742b13d3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:13:45.961457 2014 tablet_bootstrap.cc:492] T 16c9dce032904892a39472a742b13d3e P 3f8b5a5b325842fa82175c27f3eec913: Bootstrap starting.
I20260924 04:13:45.962284 2016 tablet_bootstrap.cc:492] T 16c9dce032904892a39472a742b13d3e P d0a557b2cdf74293894448646cb25674: Bootstrap starting.
I20260924 04:13:45.963511 2015 tablet_bootstrap.cc:492] T 16c9dce032904892a39472a742b13d3e P a552ee34a332411b8c448115dc3e369d: Bootstrap starting.
I20260924 04:13:45.964507 2016 tablet_bootstrap.cc:654] T 16c9dce032904892a39472a742b13d3e P d0a557b2cdf74293894448646cb25674: Neither blocks nor log segments found. Creating new log.
I20260924 04:13:45.965040 2014 tablet_bootstrap.cc:654] T 16c9dce032904892a39472a742b13d3e P 3f8b5a5b325842fa82175c27f3eec913: Neither blocks nor log segments found. Creating new log.
I20260924 04:13:45.965113 2015 tablet_bootstrap.cc:654] T 16c9dce032904892a39472a742b13d3e P a552ee34a332411b8c448115dc3e369d: Neither blocks nor log segments found. Creating new log.
I20260924 04:13:45.965386 2016 log.cc:824] T 16c9dce032904892a39472a742b13d3e P d0a557b2cdf74293894448646cb25674: Log is configured to *not* fsync() on all Append() calls
I20260924 04:13:45.965852 2014 log.cc:824] T 16c9dce032904892a39472a742b13d3e P 3f8b5a5b325842fa82175c27f3eec913: Log is configured to *not* fsync() on all Append() calls
I20260924 04:13:45.965860 2015 log.cc:824] T 16c9dce032904892a39472a742b13d3e P a552ee34a332411b8c448115dc3e369d: Log is configured to *not* fsync() on all Append() calls
I20260924 04:13:45.967449 2016 tablet_bootstrap.cc:492] T 16c9dce032904892a39472a742b13d3e P d0a557b2cdf74293894448646cb25674: No bootstrap required, opened a new log
I20260924 04:13:45.967636 2016 ts_tablet_manager.cc:1397] T 16c9dce032904892a39472a742b13d3e P d0a557b2cdf74293894448646cb25674: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:13:45.967653 2014 tablet_bootstrap.cc:492] T 16c9dce032904892a39472a742b13d3e P 3f8b5a5b325842fa82175c27f3eec913: No bootstrap required, opened a new log
I20260924 04:13:45.967818 2014 ts_tablet_manager.cc:1397] T 16c9dce032904892a39472a742b13d3e P 3f8b5a5b325842fa82175c27f3eec913: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260924 04:13:45.967948 2015 tablet_bootstrap.cc:492] T 16c9dce032904892a39472a742b13d3e P a552ee34a332411b8c448115dc3e369d: No bootstrap required, opened a new log
I20260924 04:13:45.968115 2015 ts_tablet_manager.cc:1397] T 16c9dce032904892a39472a742b13d3e P a552ee34a332411b8c448115dc3e369d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 04:13:45.970621 2014 raft_consensus.cc:348] T 16c9dce032904892a39472a742b13d3e P 3f8b5a5b325842fa82175c27f3eec913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a552ee34a332411b8c448115dc3e369d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44665 } } peers { permanent_uuid: "d0a557b2cdf74293894448646cb25674" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42723 } } peers { permanent_uuid: "3f8b5a5b325842fa82175c27f3eec913" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35997 } }
I20260924 04:13:45.970831 2014 raft_consensus.cc:374] T 16c9dce032904892a39472a742b13d3e P 3f8b5a5b325842fa82175c27f3eec913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:13:45.970888 2014 raft_consensus.cc:729] T 16c9dce032904892a39472a742b13d3e P 3f8b5a5b325842fa82175c27f3eec913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f8b5a5b325842fa82175c27f3eec913, State: Initialized, Role: FOLLOWER
I20260924 04:13:45.971328 2014 consensus_queue.cc:260] T 16c9dce032904892a39472a742b13d3e P 3f8b5a5b325842fa82175c27f3eec913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a552ee34a332411b8c448115dc3e369d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44665 } } peers { permanent_uuid: "d0a557b2cdf74293894448646cb25674" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42723 } } peers { permanent_uuid: "3f8b5a5b325842fa82175c27f3eec913" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35997 } }
I20260924 04:13:45.971454 2016 raft_consensus.cc:348] T 16c9dce032904892a39472a742b13d3e P d0a557b2cdf74293894448646cb25674 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a552ee34a332411b8c448115dc3e369d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44665 } } peers { permanent_uuid: "d0a557b2cdf74293894448646cb25674" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42723 } } peers { permanent_uuid: "3f8b5a5b325842fa82175c27f3eec913" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35997 } }
I20260924 04:13:45.971830 2016 raft_consensus.cc:374] T 16c9dce032904892a39472a742b13d3e P d0a557b2cdf74293894448646cb25674 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:13:45.971936 2016 raft_consensus.cc:729] T 16c9dce032904892a39472a742b13d3e P d0a557b2cdf74293894448646cb25674 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a557b2cdf74293894448646cb25674, State: Initialized, Role: FOLLOWER
I20260924 04:13:45.972084 1991 heartbeater.cc:503] Master 127.0.98.62:36995 was elected leader, sending a full tablet report...
I20260924 04:13:45.971685 2015 raft_consensus.cc:348] T 16c9dce032904892a39472a742b13d3e P a552ee34a332411b8c448115dc3e369d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a552ee34a332411b8c448115dc3e369d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44665 } } peers { permanent_uuid: "d0a557b2cdf74293894448646cb25674" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42723 } } peers { permanent_uuid: "3f8b5a5b325842fa82175c27f3eec913" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35997 } }
I20260924 04:13:45.972407 2015 raft_consensus.cc:374] T 16c9dce032904892a39472a742b13d3e P a552ee34a332411b8c448115dc3e369d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:13:45.972455 2015 raft_consensus.cc:729] T 16c9dce032904892a39472a742b13d3e P a552ee34a332411b8c448115dc3e369d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a552ee34a332411b8c448115dc3e369d, State: Initialized, Role: FOLLOWER
I20260924 04:13:45.972455 2014 ts_tablet_manager.cc:1428] T 16c9dce032904892a39472a742b13d3e P 3f8b5a5b325842fa82175c27f3eec913: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260924 04:13:45.972425 2016 consensus_queue.cc:260] T 16c9dce032904892a39472a742b13d3e P d0a557b2cdf74293894448646cb25674 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a552ee34a332411b8c448115dc3e369d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44665 } } peers { permanent_uuid: "d0a557b2cdf74293894448646cb25674" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42723 } } peers { permanent_uuid: "3f8b5a5b325842fa82175c27f3eec913" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35997 } }
I20260924 04:13:45.972851 2015 consensus_queue.cc:260] T 16c9dce032904892a39472a742b13d3e P a552ee34a332411b8c448115dc3e369d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a552ee34a332411b8c448115dc3e369d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44665 } } peers { permanent_uuid: "d0a557b2cdf74293894448646cb25674" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42723 } } peers { permanent_uuid: "3f8b5a5b325842fa82175c27f3eec913" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35997 } }
I20260924 04:13:45.973009 1711 heartbeater.cc:503] Master 127.0.98.62:36995 was elected leader, sending a full tablet report...
I20260924 04:13:45.973335 2016 ts_tablet_manager.cc:1428] T 16c9dce032904892a39472a742b13d3e P d0a557b2cdf74293894448646cb25674: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:13:45.973690 2015 ts_tablet_manager.cc:1428] T 16c9dce032904892a39472a742b13d3e P a552ee34a332411b8c448115dc3e369d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:13:45.973940 1851 heartbeater.cc:503] Master 127.0.98.62:36995 was elected leader, sending a full tablet report...
W20260924 04:13:45.980625 1712 tablet.cc:2369] T 16c9dce032904892a39472a742b13d3e P d0a557b2cdf74293894448646cb25674: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:13:46.118870 2021 raft_consensus.cc:482] T 16c9dce032904892a39472a742b13d3e P d0a557b2cdf74293894448646cb25674 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:13:46.119089 2021 raft_consensus.cc:504] T 16c9dce032904892a39472a742b13d3e P d0a557b2cdf74293894448646cb25674 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a552ee34a332411b8c448115dc3e369d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44665 } } peers { permanent_uuid: "d0a557b2cdf74293894448646cb25674" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42723 } } peers { permanent_uuid: "3f8b5a5b325842fa82175c27f3eec913" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35997 } }
I20260924 04:13:46.120190 2021 leader_election.cc:288] T 16c9dce032904892a39472a742b13d3e P d0a557b2cdf74293894448646cb25674 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a552ee34a332411b8c448115dc3e369d (127.0.98.2:44665), 3f8b5a5b325842fa82175c27f3eec913 (127.0.98.3:35997)
I20260924 04:13:46.124079 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:46.120440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44665 (local address 127.0.98.1:53997)
0924 04:13:46.120860 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:46.120874 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:46.120983 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:46.121217 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:46.121219 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:46.121232 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:13:46.121335 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:46.121341 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:46.122068 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:46.122072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:46.122976 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:46.122985 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:46.123600 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:46.123603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:46.123744 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:46.123764 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 04:13:46.123852 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 04:13:46.123922 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"mutex_wait_us":137,"thread_start_us":71,"threads_started":1}
I20260924 04:13:46.124131 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:46.120642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53997 (local address 127.0.98.2:44665)
0924 04:13:46.120758 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:46.120775 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:46.120960 (+ 185us) server_negotiation.cc:407] Connection header received
0924 04:13:46.121066 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:46.121069 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:46.121120 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:46.121191 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:13:46.121434 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:46.121979 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:46.123090 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:46.123496 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:46.123586 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:46.123650 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:46.123932 (+ 282us) server_negotiation.cc:299] Negotiation successful
0924 04:13:46.124007 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 04:13:46.124198 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:46.120587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59251 (local address 127.0.98.3:35997)
0924 04:13:46.120713 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:46.120717 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:46.120859 (+ 142us) server_negotiation.cc:407] Connection header received
0924 04:13:46.120972 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:46.120975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:46.121025 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:46.121095 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:13:46.121364 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:46.121901 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:46.123060 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:46.123565 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:46.123660 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:46.123720 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:46.124016 (+ 296us) server_negotiation.cc:299] Negotiation successful
0924 04:13:46.124091 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 04:13:46.124199 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:46.120440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35997 (local address 127.0.98.1:59251)
0924 04:13:46.120764 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:46.120778 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:46.120884 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:46.121110 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:46.121113 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:46.121127 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:13:46.121249 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:46.121255 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:46.122009 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:46.122012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:46.122952 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:46.122961 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:46.123665 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:46.123669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:46.123805 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:46.123863 (+ 58us) client_negotiation.cc:769] Sending connection context
0924 04:13:46.123964 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 04:13:46.124027 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":40,"thread_start_us":84,"threads_started":1}
I20260924 04:13:46.124977 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16c9dce032904892a39472a742b13d3e" candidate_uuid: "d0a557b2cdf74293894448646cb25674" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a552ee34a332411b8c448115dc3e369d" is_pre_election: true
I20260924 04:13:46.124976 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16c9dce032904892a39472a742b13d3e" candidate_uuid: "d0a557b2cdf74293894448646cb25674" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f8b5a5b325842fa82175c27f3eec913" is_pre_election: true
I20260924 04:13:46.125247 1945 raft_consensus.cc:2454] T 16c9dce032904892a39472a742b13d3e P 3f8b5a5b325842fa82175c27f3eec913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a557b2cdf74293894448646cb25674 in term 0.
I20260924 04:13:46.125255 1805 raft_consensus.cc:2454] T 16c9dce032904892a39472a742b13d3e P a552ee34a332411b8c448115dc3e369d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a557b2cdf74293894448646cb25674 in term 0.
I20260924 04:13:46.125677 1596 leader_election.cc:302] T 16c9dce032904892a39472a742b13d3e P d0a557b2cdf74293894448646cb25674 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f8b5a5b325842fa82175c27f3eec913, d0a557b2cdf74293894448646cb25674; no voters:
I20260924 04:13:46.125905 2021 raft_consensus.cc:2781] T 16c9dce032904892a39472a742b13d3e P d0a557b2cdf74293894448646cb25674 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:13:46.125985 2021 raft_consensus.cc:482] T 16c9dce032904892a39472a742b13d3e P d0a557b2cdf74293894448646cb25674 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:13:46.126032 2021 raft_consensus.cc:3037] T 16c9dce032904892a39472a742b13d3e P d0a557b2cdf74293894448646cb25674 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:13:46.126988 2021 raft_consensus.cc:504] T 16c9dce032904892a39472a742b13d3e P d0a557b2cdf74293894448646cb25674 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a552ee34a332411b8c448115dc3e369d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44665 } } peers { permanent_uuid: "d0a557b2cdf74293894448646cb25674" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42723 } } peers { permanent_uuid: "3f8b5a5b325842fa82175c27f3eec913" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35997 } }
I20260924 04:13:46.127331 2021 leader_election.cc:288] T 16c9dce032904892a39472a742b13d3e P d0a557b2cdf74293894448646cb25674 [CANDIDATE]: Term 1 election: Requested vote from peers a552ee34a332411b8c448115dc3e369d (127.0.98.2:44665), 3f8b5a5b325842fa82175c27f3eec913 (127.0.98.3:35997)
I20260924 04:13:46.127591 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16c9dce032904892a39472a742b13d3e" candidate_uuid: "d0a557b2cdf74293894448646cb25674" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a552ee34a332411b8c448115dc3e369d"
I20260924 04:13:46.127624 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16c9dce032904892a39472a742b13d3e" candidate_uuid: "d0a557b2cdf74293894448646cb25674" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f8b5a5b325842fa82175c27f3eec913"
I20260924 04:13:46.127707 1805 raft_consensus.cc:3037] T 16c9dce032904892a39472a742b13d3e P a552ee34a332411b8c448115dc3e369d [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:13:46.127737 1945 raft_consensus.cc:3037] T 16c9dce032904892a39472a742b13d3e P 3f8b5a5b325842fa82175c27f3eec913 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:13:46.128898 1945 raft_consensus.cc:2454] T 16c9dce032904892a39472a742b13d3e P 3f8b5a5b325842fa82175c27f3eec913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a557b2cdf74293894448646cb25674 in term 1.
I20260924 04:13:46.128913 1805 raft_consensus.cc:2454] T 16c9dce032904892a39472a742b13d3e P a552ee34a332411b8c448115dc3e369d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a557b2cdf74293894448646cb25674 in term 1.
I20260924 04:13:46.129182 1599 leader_election.cc:302] T 16c9dce032904892a39472a742b13d3e P d0a557b2cdf74293894448646cb25674 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a552ee34a332411b8c448115dc3e369d, d0a557b2cdf74293894448646cb25674; no voters:
I20260924 04:13:46.129333 2021 raft_consensus.cc:2781] T 16c9dce032904892a39472a742b13d3e P d0a557b2cdf74293894448646cb25674 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:13:46.129503 2021 raft_consensus.cc:686] T 16c9dce032904892a39472a742b13d3e P d0a557b2cdf74293894448646cb25674 [term 1 LEADER]: Becoming Leader. State: Replica: d0a557b2cdf74293894448646cb25674, State: Running, Role: LEADER
I20260924 04:13:46.129798 2021 consensus_queue.cc:237] T 16c9dce032904892a39472a742b13d3e P d0a557b2cdf74293894448646cb25674 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a552ee34a332411b8c448115dc3e369d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44665 } } peers { permanent_uuid: "d0a557b2cdf74293894448646cb25674" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42723 } } peers { permanent_uuid: "3f8b5a5b325842fa82175c27f3eec913" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35997 } }
I20260924 04:13:46.132715 1507 catalog_manager.cc:5495] T 16c9dce032904892a39472a742b13d3e P d0a557b2cdf74293894448646cb25674 reported cstate change: term changed from 0 to 1, leader changed from <none> to d0a557b2cdf74293894448646cb25674 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "d0a557b2cdf74293894448646cb25674" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a552ee34a332411b8c448115dc3e369d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a557b2cdf74293894448646cb25674" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42723 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f8b5a5b325842fa82175c27f3eec913" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35997 } health_report { overall_health: UNKNOWN } } }
W20260924 04:13:46.145730 1992 tablet.cc:2369] T 16c9dce032904892a39472a742b13d3e P 3f8b5a5b325842fa82175c27f3eec913: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:13:46.152952 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:46.149677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38742 (local address 127.0.98.1:42723)
0924 04:13:46.149794 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:46.149797 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:46.149810 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:13:46.149968 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:46.149970 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:46.150023 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:46.150103 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:13:46.150290 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:46.150777 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:46.151753 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:46.151973 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:46.152048 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:46.152284 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:13:46.152678 (+ 394us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:13:46.152753 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:46.152801 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 04:13:46.152847 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:13:46.158344 1945 raft_consensus.cc:1261] T 16c9dce032904892a39472a742b13d3e P 3f8b5a5b325842fa82175c27f3eec913 [term 1 FOLLOWER]: Refusing update from remote peer d0a557b2cdf74293894448646cb25674: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:13:46.158366 1805 raft_consensus.cc:1261] T 16c9dce032904892a39472a742b13d3e P a552ee34a332411b8c448115dc3e369d [term 1 FOLLOWER]: Refusing update from remote peer d0a557b2cdf74293894448646cb25674: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:13:46.158985 2025 consensus_queue.cc:1035] T 16c9dce032904892a39472a742b13d3e P d0a557b2cdf74293894448646cb25674 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a552ee34a332411b8c448115dc3e369d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:13:46.159152 2021 consensus_queue.cc:1035] T 16c9dce032904892a39472a742b13d3e P d0a557b2cdf74293894448646cb25674 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f8b5a5b325842fa82175c27f3eec913" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:13:46.167701 2030 mvcc.cc:205] Tried to move back new op lower bound from 7332754334338187264 to 7332754334230016000. Current Snapshot: MvccSnapshot[applied={T|T < 7332754334338187264}]
I20260924 04:13:46.177630 1508 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:13:46.177796 1508 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:13:46.180370 1508 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8e587b56a18342baa915c856431b902a: Sending DeleteTablet for 3 replicas of tablet 16c9dce032904892a39472a742b13d3e
I20260924 04:13:46.181226 1925 tablet_service.cc:1514] Processing DeleteTablet for tablet 16c9dce032904892a39472a742b13d3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:13:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56406
I20260924 04:13:46.181224 1645 tablet_service.cc:1514] Processing DeleteTablet for tablet 16c9dce032904892a39472a742b13d3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:13:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38730
I20260924 04:13:46.181398 1785 tablet_service.cc:1514] Processing DeleteTablet for tablet 16c9dce032904892a39472a742b13d3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:13:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55828
I20260924 04:13:46.181725 2037 tablet_replica.cc:330] T 16c9dce032904892a39472a742b13d3e P 3f8b5a5b325842fa82175c27f3eec913: stopping tablet replica
I20260924 04:13:46.181901 2039 tablet_replica.cc:330] T 16c9dce032904892a39472a742b13d3e P a552ee34a332411b8c448115dc3e369d: stopping tablet replica
I20260924 04:13:46.182081 2037 raft_consensus.cc:2229] T 16c9dce032904892a39472a742b13d3e P 3f8b5a5b325842fa82175c27f3eec913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:13:46.182258 2039 raft_consensus.cc:2229] T 16c9dce032904892a39472a742b13d3e P a552ee34a332411b8c448115dc3e369d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:13:46.182425 2037 raft_consensus.cc:2258] T 16c9dce032904892a39472a742b13d3e P 3f8b5a5b325842fa82175c27f3eec913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:13:46.182621 2039 raft_consensus.cc:2258] T 16c9dce032904892a39472a742b13d3e P a552ee34a332411b8c448115dc3e369d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:13:46.182792 2038 tablet_replica.cc:330] T 16c9dce032904892a39472a742b13d3e P d0a557b2cdf74293894448646cb25674: stopping tablet replica
I20260924 04:13:46.182742 1508 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34822:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:13:46.183156 1508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:13:46.184226 2037 ts_tablet_manager.cc:1905] T 16c9dce032904892a39472a742b13d3e P 3f8b5a5b325842fa82175c27f3eec913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:13:46.184226 2039 ts_tablet_manager.cc:1905] T 16c9dce032904892a39472a742b13d3e P a552ee34a332411b8c448115dc3e369d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:13:46.184913 2038 raft_consensus.cc:2229] T 16c9dce032904892a39472a742b13d3e P d0a557b2cdf74293894448646cb25674 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:13:46.185356 2038 raft_consensus.cc:2258] T 16c9dce032904892a39472a742b13d3e P d0a557b2cdf74293894448646cb25674 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:13:46.186653 2038 ts_tablet_manager.cc:1905] T 16c9dce032904892a39472a742b13d3e P d0a557b2cdf74293894448646cb25674: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:13:46.187288 2037 ts_tablet_manager.cc:1918] T 16c9dce032904892a39472a742b13d3e P 3f8b5a5b325842fa82175c27f3eec913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:13:46.187444 2037 log.cc:1196] T 16c9dce032904892a39472a742b13d3e P 3f8b5a5b325842fa82175c27f3eec913: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/ts-2/wal/wals/16c9dce032904892a39472a742b13d3e
I20260924 04:13:46.187824 2037 ts_tablet_manager.cc:1939] T 16c9dce032904892a39472a742b13d3e P 3f8b5a5b325842fa82175c27f3eec913: Deleting consensus metadata
I20260924 04:13:46.189224 2038 ts_tablet_manager.cc:1918] T 16c9dce032904892a39472a742b13d3e P d0a557b2cdf74293894448646cb25674: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:13:46.189219 1785 tablet_service.cc:1467] Processing CreateTablet for tablet 8a9e1a5eed0e42fcbfaeda0fcdade9a1 (DEFAULT_TABLE table=test-table [id=663f0cb1672f473784cbaf7380c99c3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:13:46.189327 2038 log.cc:1196] T 16c9dce032904892a39472a742b13d3e P d0a557b2cdf74293894448646cb25674: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/ts-0/wal/wals/16c9dce032904892a39472a742b13d3e
I20260924 04:13:46.189462 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a9e1a5eed0e42fcbfaeda0fcdade9a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:13:46.189658 2038 ts_tablet_manager.cc:1939] T 16c9dce032904892a39472a742b13d3e P d0a557b2cdf74293894448646cb25674: Deleting consensus metadata
I20260924 04:13:46.190999 1492 catalog_manager.cc:4841] TS d0a557b2cdf74293894448646cb25674 (127.0.98.1:42723): tablet 16c9dce032904892a39472a742b13d3e (table test-table [id=5f95375d40a24acf9999ac9bdf58c41f]) successfully deleted
I20260924 04:13:46.191160 1492 catalog_manager.cc:4841] TS 3f8b5a5b325842fa82175c27f3eec913 (127.0.98.3:35997): tablet 16c9dce032904892a39472a742b13d3e (table test-table [id=5f95375d40a24acf9999ac9bdf58c41f]) successfully deleted
I20260924 04:13:46.191246 1645 tablet_service.cc:1467] Processing CreateTablet for tablet 8a9e1a5eed0e42fcbfaeda0fcdade9a1 (DEFAULT_TABLE table=test-table [id=663f0cb1672f473784cbaf7380c99c3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:13:46.191493 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a9e1a5eed0e42fcbfaeda0fcdade9a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:13:46.191689 2015 tablet_bootstrap.cc:492] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P a552ee34a332411b8c448115dc3e369d: Bootstrap starting.
I20260924 04:13:46.192619 2015 tablet_bootstrap.cc:654] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P a552ee34a332411b8c448115dc3e369d: Neither blocks nor log segments found. Creating new log.
I20260924 04:13:46.193533 2039 ts_tablet_manager.cc:1918] T 16c9dce032904892a39472a742b13d3e P a552ee34a332411b8c448115dc3e369d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:13:46.193614 2039 log.cc:1196] T 16c9dce032904892a39472a742b13d3e P a552ee34a332411b8c448115dc3e369d: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/ts-1/wal/wals/16c9dce032904892a39472a742b13d3e
I20260924 04:13:46.193686 2016 tablet_bootstrap.cc:492] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P d0a557b2cdf74293894448646cb25674: Bootstrap starting.
I20260924 04:13:46.193881 2039 ts_tablet_manager.cc:1939] T 16c9dce032904892a39472a742b13d3e P a552ee34a332411b8c448115dc3e369d: Deleting consensus metadata
I20260924 04:13:46.194476 2016 tablet_bootstrap.cc:654] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P d0a557b2cdf74293894448646cb25674: Neither blocks nor log segments found. Creating new log.
I20260924 04:13:46.194986 1495 catalog_manager.cc:4841] TS a552ee34a332411b8c448115dc3e369d (127.0.98.2:44665): tablet 16c9dce032904892a39472a742b13d3e (table test-table [id=5f95375d40a24acf9999ac9bdf58c41f]) successfully deleted
I20260924 04:13:46.195210 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 8a9e1a5eed0e42fcbfaeda0fcdade9a1 (DEFAULT_TABLE table=test-table [id=663f0cb1672f473784cbaf7380c99c3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:13:46.195490 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a9e1a5eed0e42fcbfaeda0fcdade9a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:13:46.196054 2015 tablet_bootstrap.cc:492] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P a552ee34a332411b8c448115dc3e369d: No bootstrap required, opened a new log
I20260924 04:13:46.196121 2015 ts_tablet_manager.cc:1397] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P a552ee34a332411b8c448115dc3e369d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 04:13:46.196521 2015 raft_consensus.cc:348] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P a552ee34a332411b8c448115dc3e369d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0a557b2cdf74293894448646cb25674" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42723 } } peers { permanent_uuid: "a552ee34a332411b8c448115dc3e369d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44665 } } peers { permanent_uuid: "3f8b5a5b325842fa82175c27f3eec913" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35997 } }
I20260924 04:13:46.196619 2015 raft_consensus.cc:374] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P a552ee34a332411b8c448115dc3e369d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:13:46.196647 2015 raft_consensus.cc:729] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P a552ee34a332411b8c448115dc3e369d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a552ee34a332411b8c448115dc3e369d, State: Initialized, Role: FOLLOWER
I20260924 04:13:46.196766 2015 consensus_queue.cc:260] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P a552ee34a332411b8c448115dc3e369d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0a557b2cdf74293894448646cb25674" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42723 } } peers { permanent_uuid: "a552ee34a332411b8c448115dc3e369d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44665 } } peers { permanent_uuid: "3f8b5a5b325842fa82175c27f3eec913" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35997 } }
I20260924 04:13:46.197000 2015 ts_tablet_manager.cc:1428] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P a552ee34a332411b8c448115dc3e369d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 04:13:46.197611 2014 tablet_bootstrap.cc:492] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P 3f8b5a5b325842fa82175c27f3eec913: Bootstrap starting.
I20260924 04:13:46.197710 2016 tablet_bootstrap.cc:492] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P d0a557b2cdf74293894448646cb25674: No bootstrap required, opened a new log
I20260924 04:13:46.197780 2016 ts_tablet_manager.cc:1397] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P d0a557b2cdf74293894448646cb25674: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260924 04:13:46.198220 2016 raft_consensus.cc:348] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P d0a557b2cdf74293894448646cb25674 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0a557b2cdf74293894448646cb25674" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42723 } } peers { permanent_uuid: "a552ee34a332411b8c448115dc3e369d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44665 } } peers { permanent_uuid: "3f8b5a5b325842fa82175c27f3eec913" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35997 } }
I20260924 04:13:46.198315 2016 raft_consensus.cc:374] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P d0a557b2cdf74293894448646cb25674 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:13:46.198354 2016 raft_consensus.cc:729] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P d0a557b2cdf74293894448646cb25674 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a557b2cdf74293894448646cb25674, State: Initialized, Role: FOLLOWER
I20260924 04:13:46.198505 2014 tablet_bootstrap.cc:654] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P 3f8b5a5b325842fa82175c27f3eec913: Neither blocks nor log segments found. Creating new log.
I20260924 04:13:46.198470 2016 consensus_queue.cc:260] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P d0a557b2cdf74293894448646cb25674 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0a557b2cdf74293894448646cb25674" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42723 } } peers { permanent_uuid: "a552ee34a332411b8c448115dc3e369d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44665 } } peers { permanent_uuid: "3f8b5a5b325842fa82175c27f3eec913" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35997 } }
I20260924 04:13:46.198700 2016 ts_tablet_manager.cc:1428] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P d0a557b2cdf74293894448646cb25674: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:13:46.199610 2014 tablet_bootstrap.cc:492] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P 3f8b5a5b325842fa82175c27f3eec913: No bootstrap required, opened a new log
I20260924 04:13:46.199694 2014 ts_tablet_manager.cc:1397] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P 3f8b5a5b325842fa82175c27f3eec913: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 04:13:46.200114 2014 raft_consensus.cc:348] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P 3f8b5a5b325842fa82175c27f3eec913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0a557b2cdf74293894448646cb25674" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42723 } } peers { permanent_uuid: "a552ee34a332411b8c448115dc3e369d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44665 } } peers { permanent_uuid: "3f8b5a5b325842fa82175c27f3eec913" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35997 } }
I20260924 04:13:46.200217 2014 raft_consensus.cc:374] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P 3f8b5a5b325842fa82175c27f3eec913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:13:46.200249 2014 raft_consensus.cc:729] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P 3f8b5a5b325842fa82175c27f3eec913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f8b5a5b325842fa82175c27f3eec913, State: Initialized, Role: FOLLOWER
I20260924 04:13:46.200359 2014 consensus_queue.cc:260] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P 3f8b5a5b325842fa82175c27f3eec913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0a557b2cdf74293894448646cb25674" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42723 } } peers { permanent_uuid: "a552ee34a332411b8c448115dc3e369d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44665 } } peers { permanent_uuid: "3f8b5a5b325842fa82175c27f3eec913" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35997 } }
I20260924 04:13:46.200608 2014 ts_tablet_manager.cc:1428] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P 3f8b5a5b325842fa82175c27f3eec913: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:13:46.340428 2022 raft_consensus.cc:482] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P a552ee34a332411b8c448115dc3e369d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:13:46.340590 2022 raft_consensus.cc:504] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P a552ee34a332411b8c448115dc3e369d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0a557b2cdf74293894448646cb25674" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42723 } } peers { permanent_uuid: "a552ee34a332411b8c448115dc3e369d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44665 } } peers { permanent_uuid: "3f8b5a5b325842fa82175c27f3eec913" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35997 } }
I20260924 04:13:46.341609 2022 leader_election.cc:288] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P a552ee34a332411b8c448115dc3e369d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0a557b2cdf74293894448646cb25674 (127.0.98.1:42723), 3f8b5a5b325842fa82175c27f3eec913 (127.0.98.3:35997)
I20260924 04:13:46.345507 2043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:46.341879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42723 (local address 127.0.98.2:47425)
0924 04:13:46.342122 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:46.342135 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:46.342251 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:46.342591 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:46.342595 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:46.342620 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:13:46.342724 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:46.342748 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:46.343564 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:46.343568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:46.344435 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:46.344444 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:46.345100 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:46.345104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:46.345193 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:46.345236 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 04:13:46.345304 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 04:13:46.345356 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":81,"threads_started":1}
I20260924 04:13:46.345546 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:46.341950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47425 (local address 127.0.98.1:42723)
0924 04:13:46.342082 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:46.342087 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:46.342211 (+ 124us) server_negotiation.cc:407] Connection header received
0924 04:13:46.342404 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:46.342407 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:46.342457 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:46.342534 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:13:46.342873 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:46.343473 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:46.344558 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:46.344987 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:46.345064 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:46.345114 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:46.345334 (+ 220us) server_negotiation.cc:299] Negotiation successful
0924 04:13:46.345425 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:13:46.345567 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:46.342361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32817 (local address 127.0.98.3:35997)
0924 04:13:46.342616 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:46.342621 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:46.342636 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:13:46.342756 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:46.342758 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:46.342804 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:46.342879 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:13:46.343092 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:46.343641 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:46.344670 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:46.345094 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:46.345188 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:46.345243 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:46.345410 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 04:13:46.345468 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 04:13:46.345618 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:46.342250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35997 (local address 127.0.98.2:32817)
0924 04:13:46.342530 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:46.342542 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:46.342645 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:46.342884 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:46.342886 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:46.342903 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:13:46.342999 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:46.343005 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:46.343795 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:46.343797 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:46.344559 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:46.344568 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:46.345213 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:46.345216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:46.345292 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:46.345315 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 04:13:46.345387 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 04:13:46.345448 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":68,"threads_started":1}
I20260924 04:13:46.346107 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9e1a5eed0e42fcbfaeda0fcdade9a1" candidate_uuid: "a552ee34a332411b8c448115dc3e369d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f8b5a5b325842fa82175c27f3eec913" is_pre_election: true
I20260924 04:13:46.346227 1945 raft_consensus.cc:2454] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P 3f8b5a5b325842fa82175c27f3eec913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a552ee34a332411b8c448115dc3e369d in term 0.
I20260924 04:13:46.346171 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9e1a5eed0e42fcbfaeda0fcdade9a1" candidate_uuid: "a552ee34a332411b8c448115dc3e369d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a557b2cdf74293894448646cb25674" is_pre_election: true
I20260924 04:13:46.346395 1665 raft_consensus.cc:2454] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P d0a557b2cdf74293894448646cb25674 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a552ee34a332411b8c448115dc3e369d in term 0.
I20260924 04:13:46.346560 1736 leader_election.cc:302] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P a552ee34a332411b8c448115dc3e369d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f8b5a5b325842fa82175c27f3eec913, a552ee34a332411b8c448115dc3e369d; no voters:
I20260924 04:13:46.346803 2022 raft_consensus.cc:2781] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P a552ee34a332411b8c448115dc3e369d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:13:46.346861 2022 raft_consensus.cc:482] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P a552ee34a332411b8c448115dc3e369d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:13:46.346912 2022 raft_consensus.cc:3037] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P a552ee34a332411b8c448115dc3e369d [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:13:46.348069 2022 raft_consensus.cc:504] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P a552ee34a332411b8c448115dc3e369d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0a557b2cdf74293894448646cb25674" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42723 } } peers { permanent_uuid: "a552ee34a332411b8c448115dc3e369d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44665 } } peers { permanent_uuid: "3f8b5a5b325842fa82175c27f3eec913" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35997 } }
I20260924 04:13:46.348351 2022 leader_election.cc:288] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P a552ee34a332411b8c448115dc3e369d [CANDIDATE]: Term 1 election: Requested vote from peers d0a557b2cdf74293894448646cb25674 (127.0.98.1:42723), 3f8b5a5b325842fa82175c27f3eec913 (127.0.98.3:35997)
I20260924 04:13:46.348701 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9e1a5eed0e42fcbfaeda0fcdade9a1" candidate_uuid: "a552ee34a332411b8c448115dc3e369d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a557b2cdf74293894448646cb25674"
I20260924 04:13:46.348712 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9e1a5eed0e42fcbfaeda0fcdade9a1" candidate_uuid: "a552ee34a332411b8c448115dc3e369d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f8b5a5b325842fa82175c27f3eec913"
I20260924 04:13:46.348814 1665 raft_consensus.cc:3037] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P d0a557b2cdf74293894448646cb25674 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:13:46.348814 1945 raft_consensus.cc:3037] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P 3f8b5a5b325842fa82175c27f3eec913 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:13:46.349833 1945 raft_consensus.cc:2454] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P 3f8b5a5b325842fa82175c27f3eec913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a552ee34a332411b8c448115dc3e369d in term 1.
I20260924 04:13:46.349833 1665 raft_consensus.cc:2454] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P d0a557b2cdf74293894448646cb25674 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a552ee34a332411b8c448115dc3e369d in term 1.
I20260924 04:13:46.350068 1736 leader_election.cc:302] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P a552ee34a332411b8c448115dc3e369d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a552ee34a332411b8c448115dc3e369d, d0a557b2cdf74293894448646cb25674; no voters:
I20260924 04:13:46.350205 2022 raft_consensus.cc:2781] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P a552ee34a332411b8c448115dc3e369d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:13:46.350351 2022 raft_consensus.cc:686] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P a552ee34a332411b8c448115dc3e369d [term 1 LEADER]: Becoming Leader. State: Replica: a552ee34a332411b8c448115dc3e369d, State: Running, Role: LEADER
I20260924 04:13:46.350591 2022 consensus_queue.cc:237] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P a552ee34a332411b8c448115dc3e369d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0a557b2cdf74293894448646cb25674" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42723 } } peers { permanent_uuid: "a552ee34a332411b8c448115dc3e369d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44665 } } peers { permanent_uuid: "3f8b5a5b325842fa82175c27f3eec913" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35997 } }
I20260924 04:13:46.352512 1506 catalog_manager.cc:5495] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P a552ee34a332411b8c448115dc3e369d reported cstate change: term changed from 0 to 1, leader changed from <none> to a552ee34a332411b8c448115dc3e369d (127.0.98.2). New cstate: current_term: 1 leader_uuid: "a552ee34a332411b8c448115dc3e369d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0a557b2cdf74293894448646cb25674" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a552ee34a332411b8c448115dc3e369d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f8b5a5b325842fa82175c27f3eec913" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35997 } health_report { overall_health: UNKNOWN } } }
Sep 24 04:13:46 dist-test-slave-p401 krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:13:46.424515 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:46.416893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.0.1:34836)
0924 04:13:46.417146 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:46.417159 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:46.417262 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:46.417488 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:46.417490 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:46.417872 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:13:46.417975 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:46.417981 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:46.418650 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:46.418652 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:46.419305 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:46.419319 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:46.419732 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:46.419735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:46.419801 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:46.420334 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:13:46.420345 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:13:46.421679 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:13:46.423184 (+ 1505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:46.423192 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:46.423197 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:46.423457 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:46.423670 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:46.423672 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:46.423673 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:46.423753 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:46.424049 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:13:46.424053 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:13:46.424159 (+ 106us) client_negotiation.cc:769] Sending connection context
0924 04:13:46.424243 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 04:13:46.424346 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":108,"threads_started":1}
I20260924 04:13:46.424541 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:46.416931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34836 (local address 127.0.98.62:36995)
0924 04:13:46.417055 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:46.417060 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:46.417228 (+ 168us) server_negotiation.cc:407] Connection header received
0924 04:13:46.417339 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:46.417341 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:46.417394 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:46.417460 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:46.418062 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:46.418575 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:46.419410 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:46.419625 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:46.419698 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:46.421805 (+ 2107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:13:46.421808 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:13:46.421810 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:13:46.421829 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:13:46.423080 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:46.423533 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:46.423536 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:46.423537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:46.423606 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:46.423823 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:46.423825 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:46.423826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:46.423981 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:13:46.424048 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:46.424295 (+ 247us) server_negotiation.cc:299] Negotiation successful
0924 04:13:46.424407 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 04:13:46.428169 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34836:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 04:13:46.428810 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:13:46.433099 1785 tablet_service.cc:1467] Processing CreateTablet for tablet 517e34ffa88c4c3fa2510c4d9f81938a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b1aac6d4472f4cd79d87f132a67082a7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 04:13:46.433128 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 517e34ffa88c4c3fa2510c4d9f81938a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b1aac6d4472f4cd79d87f132a67082a7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 04:13:46.433128 1645 tablet_service.cc:1467] Processing CreateTablet for tablet 517e34ffa88c4c3fa2510c4d9f81938a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b1aac6d4472f4cd79d87f132a67082a7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 04:13:46.433454 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 517e34ffa88c4c3fa2510c4d9f81938a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:13:46.433485 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 517e34ffa88c4c3fa2510c4d9f81938a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:13:46.433485 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 517e34ffa88c4c3fa2510c4d9f81938a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:13:46.436136 2015 tablet_bootstrap.cc:492] T 517e34ffa88c4c3fa2510c4d9f81938a P a552ee34a332411b8c448115dc3e369d: Bootstrap starting.
I20260924 04:13:46.436151 2016 tablet_bootstrap.cc:492] T 517e34ffa88c4c3fa2510c4d9f81938a P d0a557b2cdf74293894448646cb25674: Bootstrap starting.
I20260924 04:13:46.436810 2016 tablet_bootstrap.cc:654] T 517e34ffa88c4c3fa2510c4d9f81938a P d0a557b2cdf74293894448646cb25674: Neither blocks nor log segments found. Creating new log.
I20260924 04:13:46.436877 2015 tablet_bootstrap.cc:654] T 517e34ffa88c4c3fa2510c4d9f81938a P a552ee34a332411b8c448115dc3e369d: Neither blocks nor log segments found. Creating new log.
I20260924 04:13:46.437242 2014 tablet_bootstrap.cc:492] T 517e34ffa88c4c3fa2510c4d9f81938a P 3f8b5a5b325842fa82175c27f3eec913: Bootstrap starting.
I20260924 04:13:46.437759 2016 tablet_bootstrap.cc:492] T 517e34ffa88c4c3fa2510c4d9f81938a P d0a557b2cdf74293894448646cb25674: No bootstrap required, opened a new log
I20260924 04:13:46.437831 2016 ts_tablet_manager.cc:1397] T 517e34ffa88c4c3fa2510c4d9f81938a P d0a557b2cdf74293894448646cb25674: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:13:46.437987 2015 tablet_bootstrap.cc:492] T 517e34ffa88c4c3fa2510c4d9f81938a P a552ee34a332411b8c448115dc3e369d: No bootstrap required, opened a new log
I20260924 04:13:46.438056 2015 ts_tablet_manager.cc:1397] T 517e34ffa88c4c3fa2510c4d9f81938a P a552ee34a332411b8c448115dc3e369d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 04:13:46.438088 2014 tablet_bootstrap.cc:654] T 517e34ffa88c4c3fa2510c4d9f81938a P 3f8b5a5b325842fa82175c27f3eec913: Neither blocks nor log segments found. Creating new log.
I20260924 04:13:46.438155 2016 raft_consensus.cc:348] T 517e34ffa88c4c3fa2510c4d9f81938a P d0a557b2cdf74293894448646cb25674 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0a557b2cdf74293894448646cb25674" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42723 } } peers { permanent_uuid: "a552ee34a332411b8c448115dc3e369d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44665 } } peers { permanent_uuid: "3f8b5a5b325842fa82175c27f3eec913" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35997 } }
I20260924 04:13:46.438233 2016 raft_consensus.cc:374] T 517e34ffa88c4c3fa2510c4d9f81938a P d0a557b2cdf74293894448646cb25674 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:13:46.438252 2016 raft_consensus.cc:729] T 517e34ffa88c4c3fa2510c4d9f81938a P d0a557b2cdf74293894448646cb25674 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a557b2cdf74293894448646cb25674, State: Initialized, Role: FOLLOWER
I20260924 04:13:46.438350 2016 consensus_queue.cc:260] T 517e34ffa88c4c3fa2510c4d9f81938a P d0a557b2cdf74293894448646cb25674 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0a557b2cdf74293894448646cb25674" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42723 } } peers { permanent_uuid: "a552ee34a332411b8c448115dc3e369d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44665 } } peers { permanent_uuid: "3f8b5a5b325842fa82175c27f3eec913" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35997 } }
I20260924 04:13:46.438470 2015 raft_consensus.cc:348] T 517e34ffa88c4c3fa2510c4d9f81938a P a552ee34a332411b8c448115dc3e369d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0a557b2cdf74293894448646cb25674" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42723 } } peers { permanent_uuid: "a552ee34a332411b8c448115dc3e369d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44665 } } peers { permanent_uuid: "3f8b5a5b325842fa82175c27f3eec913" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35997 } }
I20260924 04:13:46.438571 2015 raft_consensus.cc:374] T 517e34ffa88c4c3fa2510c4d9f81938a P a552ee34a332411b8c448115dc3e369d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:13:46.438576 2016 ts_tablet_manager.cc:1428] T 517e34ffa88c4c3fa2510c4d9f81938a P d0a557b2cdf74293894448646cb25674: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 04:13:46.438601 2015 raft_consensus.cc:729] T 517e34ffa88c4c3fa2510c4d9f81938a P a552ee34a332411b8c448115dc3e369d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a552ee34a332411b8c448115dc3e369d, State: Initialized, Role: FOLLOWER
I20260924 04:13:46.438544 2021 tablet_replica.cc:439] T 517e34ffa88c4c3fa2510c4d9f81938a P d0a557b2cdf74293894448646cb25674: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0a557b2cdf74293894448646cb25674" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42723 } } peers { permanent_uuid: "a552ee34a332411b8c448115dc3e369d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44665 } } peers { permanent_uuid: "3f8b5a5b325842fa82175c27f3eec913" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35997 } } }
I20260924 04:13:46.438680 2021 tablet_replica.cc:442] T 517e34ffa88c4c3fa2510c4d9f81938a P d0a557b2cdf74293894448646cb25674: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:13:46.438705 2015 consensus_queue.cc:260] T 517e34ffa88c4c3fa2510c4d9f81938a P a552ee34a332411b8c448115dc3e369d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0a557b2cdf74293894448646cb25674" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42723 } } peers { permanent_uuid: "a552ee34a332411b8c448115dc3e369d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44665 } } peers { permanent_uuid: "3f8b5a5b325842fa82175c27f3eec913" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35997 } }
I20260924 04:13:46.438946 2015 ts_tablet_manager.cc:1428] T 517e34ffa88c4c3fa2510c4d9f81938a P a552ee34a332411b8c448115dc3e369d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260924 04:13:46.439162 1852 tablet.cc:2369] T 517e34ffa88c4c3fa2510c4d9f81938a P a552ee34a332411b8c448115dc3e369d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:13:46.439750 2022 tablet_replica.cc:439] T 517e34ffa88c4c3fa2510c4d9f81938a P a552ee34a332411b8c448115dc3e369d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0a557b2cdf74293894448646cb25674" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42723 } } peers { permanent_uuid: "a552ee34a332411b8c448115dc3e369d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44665 } } peers { permanent_uuid: "3f8b5a5b325842fa82175c27f3eec913" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35997 } } }
I20260924 04:13:46.439811 2014 tablet_bootstrap.cc:492] T 517e34ffa88c4c3fa2510c4d9f81938a P 3f8b5a5b325842fa82175c27f3eec913: No bootstrap required, opened a new log
I20260924 04:13:46.439870 2022 tablet_replica.cc:442] T 517e34ffa88c4c3fa2510c4d9f81938a P a552ee34a332411b8c448115dc3e369d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:13:46.439898 2014 ts_tablet_manager.cc:1397] T 517e34ffa88c4c3fa2510c4d9f81938a P 3f8b5a5b325842fa82175c27f3eec913: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 04:13:46.440294 2014 raft_consensus.cc:348] T 517e34ffa88c4c3fa2510c4d9f81938a P 3f8b5a5b325842fa82175c27f3eec913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0a557b2cdf74293894448646cb25674" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42723 } } peers { permanent_uuid: "a552ee34a332411b8c448115dc3e369d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44665 } } peers { permanent_uuid: "3f8b5a5b325842fa82175c27f3eec913" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35997 } }
I20260924 04:13:46.440398 2014 raft_consensus.cc:374] T 517e34ffa88c4c3fa2510c4d9f81938a P 3f8b5a5b325842fa82175c27f3eec913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:13:46.440436 2014 raft_consensus.cc:729] T 517e34ffa88c4c3fa2510c4d9f81938a P 3f8b5a5b325842fa82175c27f3eec913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f8b5a5b325842fa82175c27f3eec913, State: Initialized, Role: FOLLOWER
I20260924 04:13:46.440552 2014 consensus_queue.cc:260] T 517e34ffa88c4c3fa2510c4d9f81938a P 3f8b5a5b325842fa82175c27f3eec913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0a557b2cdf74293894448646cb25674" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42723 } } peers { permanent_uuid: "a552ee34a332411b8c448115dc3e369d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44665 } } peers { permanent_uuid: "3f8b5a5b325842fa82175c27f3eec913" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35997 } }
I20260924 04:13:46.440786 2014 ts_tablet_manager.cc:1428] T 517e34ffa88c4c3fa2510c4d9f81938a P 3f8b5a5b325842fa82175c27f3eec913: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:13:46.440799 2020 tablet_replica.cc:439] T 517e34ffa88c4c3fa2510c4d9f81938a P 3f8b5a5b325842fa82175c27f3eec913: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0a557b2cdf74293894448646cb25674" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42723 } } peers { permanent_uuid: "a552ee34a332411b8c448115dc3e369d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44665 } } peers { permanent_uuid: "3f8b5a5b325842fa82175c27f3eec913" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35997 } } }
I20260924 04:13:46.440960 2020 tablet_replica.cc:442] T 517e34ffa88c4c3fa2510c4d9f81938a P 3f8b5a5b325842fa82175c27f3eec913: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:13:46.544698 2022 raft_consensus.cc:482] T 517e34ffa88c4c3fa2510c4d9f81938a P a552ee34a332411b8c448115dc3e369d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:13:46.544826 2022 raft_consensus.cc:504] T 517e34ffa88c4c3fa2510c4d9f81938a P a552ee34a332411b8c448115dc3e369d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0a557b2cdf74293894448646cb25674" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42723 } } peers { permanent_uuid: "a552ee34a332411b8c448115dc3e369d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44665 } } peers { permanent_uuid: "3f8b5a5b325842fa82175c27f3eec913" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35997 } }
I20260924 04:13:46.545176 2022 leader_election.cc:288] T 517e34ffa88c4c3fa2510c4d9f81938a P a552ee34a332411b8c448115dc3e369d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0a557b2cdf74293894448646cb25674 (127.0.98.1:42723), 3f8b5a5b325842fa82175c27f3eec913 (127.0.98.3:35997)
I20260924 04:13:46.545655 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "517e34ffa88c4c3fa2510c4d9f81938a" candidate_uuid: "a552ee34a332411b8c448115dc3e369d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f8b5a5b325842fa82175c27f3eec913" is_pre_election: true
I20260924 04:13:46.545646 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "517e34ffa88c4c3fa2510c4d9f81938a" candidate_uuid: "a552ee34a332411b8c448115dc3e369d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a557b2cdf74293894448646cb25674" is_pre_election: true
I20260924 04:13:46.545804 1665 raft_consensus.cc:2454] T 517e34ffa88c4c3fa2510c4d9f81938a P d0a557b2cdf74293894448646cb25674 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a552ee34a332411b8c448115dc3e369d in term 0.
I20260924 04:13:46.545804 1945 raft_consensus.cc:2454] T 517e34ffa88c4c3fa2510c4d9f81938a P 3f8b5a5b325842fa82175c27f3eec913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a552ee34a332411b8c448115dc3e369d in term 0.
I20260924 04:13:46.546077 1736 leader_election.cc:302] T 517e34ffa88c4c3fa2510c4d9f81938a P a552ee34a332411b8c448115dc3e369d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a552ee34a332411b8c448115dc3e369d, d0a557b2cdf74293894448646cb25674; no voters:
I20260924 04:13:46.546239 2022 raft_consensus.cc:2781] T 517e34ffa88c4c3fa2510c4d9f81938a P a552ee34a332411b8c448115dc3e369d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:13:46.546300 2022 raft_consensus.cc:482] T 517e34ffa88c4c3fa2510c4d9f81938a P a552ee34a332411b8c448115dc3e369d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:13:46.546326 2022 raft_consensus.cc:3037] T 517e34ffa88c4c3fa2510c4d9f81938a P a552ee34a332411b8c448115dc3e369d [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:13:46.547252 2022 raft_consensus.cc:504] T 517e34ffa88c4c3fa2510c4d9f81938a P a552ee34a332411b8c448115dc3e369d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0a557b2cdf74293894448646cb25674" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42723 } } peers { permanent_uuid: "a552ee34a332411b8c448115dc3e369d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44665 } } peers { permanent_uuid: "3f8b5a5b325842fa82175c27f3eec913" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35997 } }
I20260924 04:13:46.547547 2022 leader_election.cc:288] T 517e34ffa88c4c3fa2510c4d9f81938a P a552ee34a332411b8c448115dc3e369d [CANDIDATE]: Term 1 election: Requested vote from peers d0a557b2cdf74293894448646cb25674 (127.0.98.1:42723), 3f8b5a5b325842fa82175c27f3eec913 (127.0.98.3:35997)
I20260924 04:13:46.547845 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "517e34ffa88c4c3fa2510c4d9f81938a" candidate_uuid: "a552ee34a332411b8c448115dc3e369d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a557b2cdf74293894448646cb25674"
I20260924 04:13:46.547956 1665 raft_consensus.cc:3037] T 517e34ffa88c4c3fa2510c4d9f81938a P d0a557b2cdf74293894448646cb25674 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:13:46.547925 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "517e34ffa88c4c3fa2510c4d9f81938a" candidate_uuid: "a552ee34a332411b8c448115dc3e369d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f8b5a5b325842fa82175c27f3eec913"
I20260924 04:13:46.548044 1945 raft_consensus.cc:3037] T 517e34ffa88c4c3fa2510c4d9f81938a P 3f8b5a5b325842fa82175c27f3eec913 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:13:46.548827 1665 raft_consensus.cc:2454] T 517e34ffa88c4c3fa2510c4d9f81938a P d0a557b2cdf74293894448646cb25674 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a552ee34a332411b8c448115dc3e369d in term 1.
I20260924 04:13:46.548866 1945 raft_consensus.cc:2454] T 517e34ffa88c4c3fa2510c4d9f81938a P 3f8b5a5b325842fa82175c27f3eec913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a552ee34a332411b8c448115dc3e369d in term 1.
I20260924 04:13:46.549073 1736 leader_election.cc:302] T 517e34ffa88c4c3fa2510c4d9f81938a P a552ee34a332411b8c448115dc3e369d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a552ee34a332411b8c448115dc3e369d, d0a557b2cdf74293894448646cb25674; no voters:
I20260924 04:13:46.549227 2022 raft_consensus.cc:2781] T 517e34ffa88c4c3fa2510c4d9f81938a P a552ee34a332411b8c448115dc3e369d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:13:46.549304 2022 raft_consensus.cc:686] T 517e34ffa88c4c3fa2510c4d9f81938a P a552ee34a332411b8c448115dc3e369d [term 1 LEADER]: Becoming Leader. State: Replica: a552ee34a332411b8c448115dc3e369d, State: Running, Role: LEADER
I20260924 04:13:46.549463 2022 consensus_queue.cc:237] T 517e34ffa88c4c3fa2510c4d9f81938a P a552ee34a332411b8c448115dc3e369d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0a557b2cdf74293894448646cb25674" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42723 } } peers { permanent_uuid: "a552ee34a332411b8c448115dc3e369d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44665 } } peers { permanent_uuid: "3f8b5a5b325842fa82175c27f3eec913" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35997 } }
I20260924 04:13:46.550082 2045 tablet_replica.cc:439] T 517e34ffa88c4c3fa2510c4d9f81938a P a552ee34a332411b8c448115dc3e369d: TxnStatusTablet state changed. Reason: New leader a552ee34a332411b8c448115dc3e369d. Latest consensus state: current_term: 1 leader_uuid: "a552ee34a332411b8c448115dc3e369d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0a557b2cdf74293894448646cb25674" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42723 } } peers { permanent_uuid: "a552ee34a332411b8c448115dc3e369d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44665 } } peers { permanent_uuid: "3f8b5a5b325842fa82175c27f3eec913" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35997 } } }
I20260924 04:13:46.550184 2045 tablet_replica.cc:442] T 517e34ffa88c4c3fa2510c4d9f81938a P a552ee34a332411b8c448115dc3e369d: This TxnStatusTablet replica's current role is: LEADER
I20260924 04:13:46.550432 2057 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 04:13:46.550578 2057 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 04:13:46.551041 1506 catalog_manager.cc:5495] T 517e34ffa88c4c3fa2510c4d9f81938a P a552ee34a332411b8c448115dc3e369d reported cstate change: term changed from 0 to 1, leader changed from <none> to a552ee34a332411b8c448115dc3e369d (127.0.98.2). New cstate: current_term: 1 leader_uuid: "a552ee34a332411b8c448115dc3e369d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0a557b2cdf74293894448646cb25674" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a552ee34a332411b8c448115dc3e369d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f8b5a5b325842fa82175c27f3eec913" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35997 } health_report { overall_health: UNKNOWN } } }
I20260924 04:13:46.563331 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:46.559934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44665 (local address 127.0.0.1:55836)
0924 04:13:46.560076 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:46.560091 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:46.560189 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:46.560486 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:46.560489 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:46.560498 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:13:46.560593 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:46.560599 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:46.561271 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:46.561275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:46.561868 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:46.561885 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:46.562278 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:46.562281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:46.562362 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:46.562404 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:13:46.562989 (+ 585us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:13:46.563002 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 04:13:46.563099 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 04:13:46.563184 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 04:13:46.563331 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:46.560015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55836 (local address 127.0.98.2:44665)
0924 04:13:46.560112 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:46.560116 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:46.560129 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:13:46.560316 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:46.560320 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:46.560382 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:46.560462 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:13:46.560687 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:46.561176 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:46.561981 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:46.562190 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:46.562271 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:46.562530 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:13:46.562901 (+ 371us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:13:46.562997 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:46.563118 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 04:13:46.563169 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 04:13:46.566325 1945 raft_consensus.cc:1261] T 517e34ffa88c4c3fa2510c4d9f81938a P 3f8b5a5b325842fa82175c27f3eec913 [term 1 FOLLOWER]: Refusing update from remote peer a552ee34a332411b8c448115dc3e369d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:13:46.566527 1665 raft_consensus.cc:1261] T 517e34ffa88c4c3fa2510c4d9f81938a P d0a557b2cdf74293894448646cb25674 [term 1 FOLLOWER]: Refusing update from remote peer a552ee34a332411b8c448115dc3e369d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:13:46.566792 2045 consensus_queue.cc:1035] T 517e34ffa88c4c3fa2510c4d9f81938a P a552ee34a332411b8c448115dc3e369d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f8b5a5b325842fa82175c27f3eec913" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:13:46.567020 2022 consensus_queue.cc:1035] T 517e34ffa88c4c3fa2510c4d9f81938a P a552ee34a332411b8c448115dc3e369d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a557b2cdf74293894448646cb25674" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:13:46.568497 2020 tablet_replica.cc:439] T 517e34ffa88c4c3fa2510c4d9f81938a P 3f8b5a5b325842fa82175c27f3eec913: TxnStatusTablet state changed. Reason: New leader a552ee34a332411b8c448115dc3e369d. Latest consensus state: current_term: 1 leader_uuid: "a552ee34a332411b8c448115dc3e369d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0a557b2cdf74293894448646cb25674" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42723 } } peers { permanent_uuid: "a552ee34a332411b8c448115dc3e369d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44665 } } peers { permanent_uuid: "3f8b5a5b325842fa82175c27f3eec913" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35997 } } }
I20260924 04:13:46.568609 2020 tablet_replica.cc:442] T 517e34ffa88c4c3fa2510c4d9f81938a P 3f8b5a5b325842fa82175c27f3eec913: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:13:46.568657 2021 tablet_replica.cc:439] T 517e34ffa88c4c3fa2510c4d9f81938a P d0a557b2cdf74293894448646cb25674: TxnStatusTablet state changed. Reason: New leader a552ee34a332411b8c448115dc3e369d. Latest consensus state: current_term: 1 leader_uuid: "a552ee34a332411b8c448115dc3e369d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0a557b2cdf74293894448646cb25674" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42723 } } peers { permanent_uuid: "a552ee34a332411b8c448115dc3e369d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44665 } } peers { permanent_uuid: "3f8b5a5b325842fa82175c27f3eec913" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35997 } } }
I20260924 04:13:46.568756 2021 tablet_replica.cc:442] T 517e34ffa88c4c3fa2510c4d9f81938a P d0a557b2cdf74293894448646cb25674: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:13:46.570015 2045 tablet_replica.cc:439] T 517e34ffa88c4c3fa2510c4d9f81938a P a552ee34a332411b8c448115dc3e369d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a552ee34a332411b8c448115dc3e369d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0a557b2cdf74293894448646cb25674" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42723 } } peers { permanent_uuid: "a552ee34a332411b8c448115dc3e369d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44665 } } peers { permanent_uuid: "3f8b5a5b325842fa82175c27f3eec913" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35997 } } }
I20260924 04:13:46.570120 2045 tablet_replica.cc:442] T 517e34ffa88c4c3fa2510c4d9f81938a P a552ee34a332411b8c448115dc3e369d: This TxnStatusTablet replica's current role is: LEADER
I20260924 04:13:46.570088 2029 mvcc.cc:205] Tried to move back new op lower bound from 7332754336010985472 to 7332754335947776000. Current Snapshot: MvccSnapshot[applied={T|T < 7332754336010985472}]
I20260924 04:13:46.571333 2020 tablet_replica.cc:439] T 517e34ffa88c4c3fa2510c4d9f81938a P 3f8b5a5b325842fa82175c27f3eec913: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a552ee34a332411b8c448115dc3e369d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0a557b2cdf74293894448646cb25674" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42723 } } peers { permanent_uuid: "a552ee34a332411b8c448115dc3e369d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44665 } } peers { permanent_uuid: "3f8b5a5b325842fa82175c27f3eec913" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35997 } } }
I20260924 04:13:46.571465 2020 tablet_replica.cc:442] T 517e34ffa88c4c3fa2510c4d9f81938a P 3f8b5a5b325842fa82175c27f3eec913: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:13:46.577725 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:46.574624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55846 (local address 127.0.98.2:44665)
0924 04:13:46.574720 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:46.574723 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:46.574788 (+ 65us) server_negotiation.cc:407] Connection header received
0924 04:13:46.574919 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:46.574921 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:46.574959 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:46.575007 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:13:46.575223 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:46.575745 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:46.576862 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:46.577068 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:46.577138 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:46.577379 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:13:46.577438 (+ 59us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:13:46.577483 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:46.577575 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 04:13:46.577611 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 04:13:46.580492 2022 tablet_replica.cc:439] T 517e34ffa88c4c3fa2510c4d9f81938a P a552ee34a332411b8c448115dc3e369d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a552ee34a332411b8c448115dc3e369d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0a557b2cdf74293894448646cb25674" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42723 } } peers { permanent_uuid: "a552ee34a332411b8c448115dc3e369d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44665 } } peers { permanent_uuid: "3f8b5a5b325842fa82175c27f3eec913" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35997 } } }
I20260924 04:13:46.580605 2022 tablet_replica.cc:442] T 517e34ffa88c4c3fa2510c4d9f81938a P a552ee34a332411b8c448115dc3e369d: This TxnStatusTablet replica's current role is: LEADER
I20260924 04:13:46.581691 2028 mvcc.cc:205] Tried to move back new op lower bound from 7332754336010985472 to 7332754335947776000. Current Snapshot: MvccSnapshot[applied={T|T < 7332754336010985472}]
I20260924 04:13:46.582574 2021 tablet_replica.cc:439] T 517e34ffa88c4c3fa2510c4d9f81938a P d0a557b2cdf74293894448646cb25674: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a552ee34a332411b8c448115dc3e369d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0a557b2cdf74293894448646cb25674" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42723 } } peers { permanent_uuid: "a552ee34a332411b8c448115dc3e369d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44665 } } peers { permanent_uuid: "3f8b5a5b325842fa82175c27f3eec913" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35997 } } }
I20260924 04:13:46.582676 2021 tablet_replica.cc:442] T 517e34ffa88c4c3fa2510c4d9f81938a P d0a557b2cdf74293894448646cb25674: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:13:46.589638 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:46.586255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48503 (local address 127.0.98.2:44665)
0924 04:13:46.586362 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:46.586367 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:46.586556 (+ 189us) server_negotiation.cc:407] Connection header received
0924 04:13:46.586692 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:46.586694 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:46.586732 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:46.586788 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:13:46.587010 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:46.587567 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:46.588766 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:46.588966 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:46.589033 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:46.589292 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:13:46.589349 (+ 57us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:13:46.589398 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:46.589488 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 04:13:46.589532 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 04:13:46.589735 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:46.586210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44665 (local address 127.0.98.2:48503)
0924 04:13:46.586489 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:46.586504 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:46.586616 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:46.586806 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:46.586809 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:46.586815 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:13:46.586934 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:46.586940 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:46.587643 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:46.587646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:46.588654 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:46.588664 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:46.589050 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:46.589054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:46.589139 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:46.589187 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:13:46.589413 (+ 226us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:13:46.589419 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 04:13:46.589470 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 04:13:46.589543 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":94,"threads_started":1}
I20260924 04:13:46.595708 1665 raft_consensus.cc:1261] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P d0a557b2cdf74293894448646cb25674 [term 1 FOLLOWER]: Refusing update from remote peer a552ee34a332411b8c448115dc3e369d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:13:46.595753 1945 raft_consensus.cc:1261] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P 3f8b5a5b325842fa82175c27f3eec913 [term 1 FOLLOWER]: Refusing update from remote peer a552ee34a332411b8c448115dc3e369d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:13:46.596109 2045 consensus_queue.cc:1035] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P a552ee34a332411b8c448115dc3e369d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f8b5a5b325842fa82175c27f3eec913" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:13:46.596246 2022 consensus_queue.cc:1035] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P a552ee34a332411b8c448115dc3e369d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a557b2cdf74293894448646cb25674" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:13:46.634246 1506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:13:46.634413 1506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:13:46.636250 1506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8e587b56a18342baa915c856431b902a: Sending DeleteTablet for 3 replicas of tablet 8a9e1a5eed0e42fcbfaeda0fcdade9a1
I20260924 04:13:46.636868 1925 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a9e1a5eed0e42fcbfaeda0fcdade9a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:13:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56406
I20260924 04:13:46.636874 1785 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a9e1a5eed0e42fcbfaeda0fcdade9a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:13:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55828
I20260924 04:13:46.637048 2039 tablet_replica.cc:330] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P a552ee34a332411b8c448115dc3e369d: stopping tablet replica
I20260924 04:13:46.637154 2039 raft_consensus.cc:2229] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P a552ee34a332411b8c448115dc3e369d [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:13:46.637378 2039 raft_consensus.cc:2258] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P a552ee34a332411b8c448115dc3e369d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:13:46.637693 2037 tablet_replica.cc:330] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P 3f8b5a5b325842fa82175c27f3eec913: stopping tablet replica
I20260924 04:13:46.637894 1645 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a9e1a5eed0e42fcbfaeda0fcdade9a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:13:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38730
I20260924 04:13:46.638051 2038 tablet_replica.cc:330] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P d0a557b2cdf74293894448646cb25674: stopping tablet replica
I20260924 04:13:46.638067 2037 raft_consensus.cc:2229] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P 3f8b5a5b325842fa82175c27f3eec913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:13:46.638132 2039 ts_tablet_manager.cc:1905] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P a552ee34a332411b8c448115dc3e369d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:13:46.638160 2037 raft_consensus.cc:2258] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P 3f8b5a5b325842fa82175c27f3eec913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:13:46.638155 2038 raft_consensus.cc:2229] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P d0a557b2cdf74293894448646cb25674 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:13:46.638252 2038 raft_consensus.cc:2258] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P d0a557b2cdf74293894448646cb25674 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:13:46.638949 2037 ts_tablet_manager.cc:1905] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P 3f8b5a5b325842fa82175c27f3eec913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:13:46.639034 2038 ts_tablet_manager.cc:1905] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P d0a557b2cdf74293894448646cb25674: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:13:46.640933 2039 ts_tablet_manager.cc:1918] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P a552ee34a332411b8c448115dc3e369d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:13:46.641004 2039 log.cc:1196] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P a552ee34a332411b8c448115dc3e369d: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/ts-1/wal/wals/8a9e1a5eed0e42fcbfaeda0fcdade9a1
I20260924 04:13:46.641227 2039 ts_tablet_manager.cc:1939] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P a552ee34a332411b8c448115dc3e369d: Deleting consensus metadata
I20260924 04:13:46.641530 2038 ts_tablet_manager.cc:1918] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P d0a557b2cdf74293894448646cb25674: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:13:46.641601 2038 log.cc:1196] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P d0a557b2cdf74293894448646cb25674: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/ts-0/wal/wals/8a9e1a5eed0e42fcbfaeda0fcdade9a1
I20260924 04:13:46.641698 2037 ts_tablet_manager.cc:1918] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P 3f8b5a5b325842fa82175c27f3eec913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:13:46.641753 2037 log.cc:1196] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P 3f8b5a5b325842fa82175c27f3eec913: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790223216265354-392-0/minicluster-data/ts-2/wal/wals/8a9e1a5eed0e42fcbfaeda0fcdade9a1
I20260924 04:13:46.641810 2038 ts_tablet_manager.cc:1939] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P d0a557b2cdf74293894448646cb25674: Deleting consensus metadata
I20260924 04:13:46.641857 1495 catalog_manager.cc:4841] TS a552ee34a332411b8c448115dc3e369d (127.0.98.2:44665): tablet 8a9e1a5eed0e42fcbfaeda0fcdade9a1 (table test-table [id=663f0cb1672f473784cbaf7380c99c3a]) successfully deleted
I20260924 04:13:46.641978 2037 ts_tablet_manager.cc:1939] T 8a9e1a5eed0e42fcbfaeda0fcdade9a1 P 3f8b5a5b325842fa82175c27f3eec913: Deleting consensus metadata
I20260924 04:13:46.642445 1492 catalog_manager.cc:4841] TS d0a557b2cdf74293894448646cb25674 (127.0.98.1:42723): tablet 8a9e1a5eed0e42fcbfaeda0fcdade9a1 (table test-table [id=663f0cb1672f473784cbaf7380c99c3a]) successfully deleted
I20260924 04:13:46.642581 1492 catalog_manager.cc:4841] TS 3f8b5a5b325842fa82175c27f3eec913 (127.0.98.3:35997): tablet 8a9e1a5eed0e42fcbfaeda0fcdade9a1 (table test-table [id=663f0cb1672f473784cbaf7380c99c3a]) successfully deleted
Sep 24 04:13:46 dist-test-slave-p401 krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223225, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:13:46.652362 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:46.642267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38746 (local address 127.0.98.1:42723)
0924 04:13:46.642403 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:46.642408 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:46.643001 (+ 593us) server_negotiation.cc:407] Connection header received
0924 04:13:46.643080 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:46.643083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:46.643128 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:46.643217 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:46.643889 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:46.644715 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:46.645564 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:46.645891 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:46.645988 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:46.649205 (+ 3217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:13:46.649213 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:13:46.649227 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:13:46.649249 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:13:46.650750 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:46.651177 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:46.651182 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:46.651183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:46.651288 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:46.651553 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:46.651555 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:46.651556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:46.651780 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:13:46.651929 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:46.652148 (+ 219us) server_negotiation.cc:299] Negotiation successful
0924 04:13:46.652239 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260924 04:13:46.652786 1705 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38746
I20260924 04:13:46.661985 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:46.654411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34850 (local address 127.0.98.62:36995)
0924 04:13:46.654543 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:46.654547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:46.654562 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:13:46.654658 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:46.654661 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:46.654708 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:46.654784 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:46.655327 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:46.656107 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:46.656946 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:46.657265 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:46.657369 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:46.658830 (+ 1461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:13:46.658833 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:13:46.658835 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:13:46.658860 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:13:46.660513 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:46.660956 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:46.660958 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:46.660960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:46.661032 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:46.661272 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:46.661274 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:46.661276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:46.661448 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:13:46.661539 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:46.661765 (+ 226us) server_negotiation.cc:299] Negotiation successful
0924 04:13:46.661865 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260924 04:13:46.662364 1506 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34850
I20260924 04:13:46.663636 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 1580
I20260924 04:13:46.671629 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 1720
I20260924 04:13:46.677359 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 1860
I20260924 04:13:46.681752 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 1476
2026-09-24T04:13:46Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2693 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:13:46 dist-test-slave-p401 krb5kdc[2076](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:13:46 dist-test-slave-p401 krb5kdc[2076](info): set up 2 sockets
Sep 24 04:13:46 dist-test-slave-p401 krb5kdc[2076](info): commencing operation
krb5kdc: starting...
W20260924 04:13:47.626838 392 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/krb5kdc/test-user.keytab.
Sep 24 04:13:47 dist-test-slave-p401 krb5kdc[2076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:13:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:13:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:13:47.740229 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39493
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39493
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:13:47.844895 2089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:13:47.845149 2089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:13:47.845187 2089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:13:47.849078 2089 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:13:47.849129 2089 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 04:13:47.849144 2089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:13:47.849161 2089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:13:47.849175 2089 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:13:47.849191 2089 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:13:47.854233 2089 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39493
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:13:47.855131 2089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:13:47.856081 2089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:13:47.863898 2095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:13:47.863907 2094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:13:47.863951 2097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:13:47.864362 2089 server_base.cc:1034] running on GCE node
I20260924 04:13:47.864900 2089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:13:47.865952 2089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:13:47.867105 2089 hybrid_clock.cc:648] HybridClock initialized: now 1790223227867089 us; error 28 us; skew 500 ppm
Sep 24 04:13:47 dist-test-slave-p401 krb5kdc[2076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223227, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:13:47.869608 2089 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:13:47.870453 2089 webserver.cc:460] Webserver started at http://127.0.98.62:43591/ using document root <none> and password file <none>
I20260924 04:13:47.870975 2089 fs_manager.cc:359] Metadata directory not provided
I20260924 04:13:47.871037 2089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:13:47.871212 2089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:13:47.873183 2089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/master-0/data/instance:
uuid: "2fac17aff8b64408b4db746cefeeddbc"
format_stamp: "Formatted at 2026-09-24 04:13:47 on dist-test-slave-p401"
server_key: "9658d479e6f439bc81cd8d7a362bf724"
server_key_iv: "76ad4e3f962cde42e5703bd3d060ebb2"
server_key_version: "encryptionkey@0"
I20260924 04:13:47.873660 2089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/master-0/wal/instance:
uuid: "2fac17aff8b64408b4db746cefeeddbc"
format_stamp: "Formatted at 2026-09-24 04:13:47 on dist-test-slave-p401"
server_key: "9658d479e6f439bc81cd8d7a362bf724"
server_key_iv: "76ad4e3f962cde42e5703bd3d060ebb2"
server_key_version: "encryptionkey@0"
I20260924 04:13:47.877074 2089 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:13:47.879200 2104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:13:47.880369 2089 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:13:47.880457 2089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/master-0/wal
uuid: "2fac17aff8b64408b4db746cefeeddbc"
format_stamp: "Formatted at 2026-09-24 04:13:47 on dist-test-slave-p401"
server_key: "9658d479e6f439bc81cd8d7a362bf724"
server_key_iv: "76ad4e3f962cde42e5703bd3d060ebb2"
server_key_version: "encryptionkey@0"
I20260924 04:13:47.880535 2089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:13:47.897140 2089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:13:47.900250 2089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:13:47.900426 2089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:13:47.907516 2089 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39493
I20260924 04:13:47.907548 2166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39493 every 8 connection(s)
I20260924 04:13:47.908384 2089 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:13:47.910976 2167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:13:47.915513 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 2089
I20260924 04:13:47.915613 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/master-0/wal/instance
I20260924 04:13:47.915841 392 external_mini_cluster.cc:1428] Setting key bc72fe53ccde1396abe7a7501c01dd0e
I20260924 04:13:47.916210 2167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fac17aff8b64408b4db746cefeeddbc: Bootstrap starting.
I20260924 04:13:47.918391 2167 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2fac17aff8b64408b4db746cefeeddbc: Neither blocks nor log segments found. Creating new log.
I20260924 04:13:47.919155 2167 log.cc:824] T 00000000000000000000000000000000 P 2fac17aff8b64408b4db746cefeeddbc: Log is configured to *not* fsync() on all Append() calls
I20260924 04:13:47.920971 2167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fac17aff8b64408b4db746cefeeddbc: No bootstrap required, opened a new log
I20260924 04:13:47.924242 2167 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2fac17aff8b64408b4db746cefeeddbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fac17aff8b64408b4db746cefeeddbc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39493 } }
I20260924 04:13:47.924503 2167 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2fac17aff8b64408b4db746cefeeddbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:13:47.924587 2167 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2fac17aff8b64408b4db746cefeeddbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fac17aff8b64408b4db746cefeeddbc, State: Initialized, Role: FOLLOWER
Sep 24 04:13:47 dist-test-slave-p401 krb5kdc[2076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:13:47.925034 2167 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2fac17aff8b64408b4db746cefeeddbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fac17aff8b64408b4db746cefeeddbc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39493 } }
I20260924 04:13:47.925185 2167 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2fac17aff8b64408b4db746cefeeddbc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:13:47.925251 2167 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2fac17aff8b64408b4db746cefeeddbc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:13:47.925345 2167 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2fac17aff8b64408b4db746cefeeddbc [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:13:47.926348 2167 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2fac17aff8b64408b4db746cefeeddbc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fac17aff8b64408b4db746cefeeddbc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39493 } }
I20260924 04:13:47.926662 2167 leader_election.cc:302] T 00000000000000000000000000000000 P 2fac17aff8b64408b4db746cefeeddbc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2fac17aff8b64408b4db746cefeeddbc; no voters:
I20260924 04:13:47.926934 2167 leader_election.cc:288] T 00000000000000000000000000000000 P 2fac17aff8b64408b4db746cefeeddbc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:13:47.927033 2172 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2fac17aff8b64408b4db746cefeeddbc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:13:47.927273 2172 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2fac17aff8b64408b4db746cefeeddbc [term 1 LEADER]: Becoming Leader. State: Replica: 2fac17aff8b64408b4db746cefeeddbc, State: Running, Role: LEADER
I20260924 04:13:47.927600 2172 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2fac17aff8b64408b4db746cefeeddbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fac17aff8b64408b4db746cefeeddbc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39493 } }
I20260924 04:13:47.928148 2167 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2fac17aff8b64408b4db746cefeeddbc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:13:47.929100 2174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fac17aff8b64408b4db746cefeeddbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2fac17aff8b64408b4db746cefeeddbc. Latest consensus state: current_term: 1 leader_uuid: "2fac17aff8b64408b4db746cefeeddbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fac17aff8b64408b4db746cefeeddbc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39493 } } }
I20260924 04:13:47.929214 2174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fac17aff8b64408b4db746cefeeddbc [sys.catalog]: This master's current role is: LEADER
I20260924 04:13:47.929555 2181 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:13:47.929528 2173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fac17aff8b64408b4db746cefeeddbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2fac17aff8b64408b4db746cefeeddbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fac17aff8b64408b4db746cefeeddbc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39493 } } }
I20260924 04:13:47.929647 2173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fac17aff8b64408b4db746cefeeddbc [sys.catalog]: This master's current role is: LEADER
I20260924 04:13:47.931139 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:47.917105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50530 (local address 127.0.98.62:39493)
0924 04:13:47.917561 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:47.917576 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:47.917611 (+ 35us) server_negotiation.cc:407] Connection header received
0924 04:13:47.918232 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:47.918247 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:47.918605 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:47.918892 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:47.919795 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:47.920797 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:47.921938 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:47.922840 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:47.922972 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:47.925602 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:13:47.925608 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:13:47.925624 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:13:47.925651 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:13:47.927389 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:47.927826 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:47.927831 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:47.927837 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:47.927971 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:47.928290 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:47.928294 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:47.928296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:47.928785 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:13:47.928973 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:47.929792 (+ 819us) server_negotiation.cc:299] Negotiation successful
0924 04:13:47.930089 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":148,"threads_started":1}
I20260924 04:13:47.932701 2181 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:13:47.937978 2181 catalog_manager.cc:1347] Generated new cluster ID: 18c6f32f8bf841028b205f82f6222569
I20260924 04:13:47.938030 2181 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:13:47.950278 2181 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:13:47.951201 2181 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:13:47.956058 2181 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2fac17aff8b64408b4db746cefeeddbc: Generated new TSK 0
I20260924 04:13:47.956674 2181 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:13:47.986079 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39493
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:13:48.113080 2193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:13:48.113336 2193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:13:48.113405 2193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:13:48.117383 2193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:13:48.117451 2193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:13:48.117494 2193 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 04:13:48.117517 2193 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 04:13:48.117569 2193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:13:48.122839 2193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:13:48.123891 2193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:13:48.124891 2193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:13:48.134224 2198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:13:48.134253 2199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:13:48.134311 2201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:13:48.134276 2193 server_base.cc:1034] running on GCE node
I20260924 04:13:48.134894 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:13:48.135557 2193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:13:48.136745 2193 hybrid_clock.cc:648] HybridClock initialized: now 1790223228136723 us; error 31 us; skew 500 ppm
Sep 24 04:13:48 dist-test-slave-p401 krb5kdc[2076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223228, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:13:48.139438 2193 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:13:48.140221 2193 webserver.cc:460] Webserver started at http://127.0.98.1:34099/ using document root <none> and password file <none>
I20260924 04:13:48.140722 2193 fs_manager.cc:359] Metadata directory not provided
I20260924 04:13:48.140775 2193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:13:48.140936 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:13:48.142835 2193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/ts-0/data/instance:
uuid: "fcdc2242e68749518a954101b9abc0f0"
format_stamp: "Formatted at 2026-09-24 04:13:48 on dist-test-slave-p401"
server_key: "6704b45cd86cbd0c99dc47d8f41ca4c3"
server_key_iv: "5dd8ffe40f757df2b4229d8118772ea7"
server_key_version: "encryptionkey@0"
I20260924 04:13:48.143374 2193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/ts-0/wal/instance:
uuid: "fcdc2242e68749518a954101b9abc0f0"
format_stamp: "Formatted at 2026-09-24 04:13:48 on dist-test-slave-p401"
server_key: "6704b45cd86cbd0c99dc47d8f41ca4c3"
server_key_iv: "5dd8ffe40f757df2b4229d8118772ea7"
server_key_version: "encryptionkey@0"
I20260924 04:13:48.146714 2193 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:13:48.148926 2208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:13:48.150074 2193 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:13:48.150168 2193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/ts-0/wal
uuid: "fcdc2242e68749518a954101b9abc0f0"
format_stamp: "Formatted at 2026-09-24 04:13:48 on dist-test-slave-p401"
server_key: "6704b45cd86cbd0c99dc47d8f41ca4c3"
server_key_iv: "5dd8ffe40f757df2b4229d8118772ea7"
server_key_version: "encryptionkey@0"
I20260924 04:13:48.150265 2193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:13:48.161670 2193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:13:48.164379 2193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:13:48.164543 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:13:48.165879 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:13:48.165926 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:13:48.165966 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:13:48.165994 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 04:13:48 dist-test-slave-p401 krb5kdc[2076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223228, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:13:48.175671 2193 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38113
I20260924 04:13:48.175702 2323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38113 every 8 connection(s)
I20260924 04:13:48.176810 2193 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:13:48.178933 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:48.166604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55187 (local address 127.0.98.62:39493)
0924 04:13:48.166752 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:48.166758 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:48.167577 (+ 819us) server_negotiation.cc:407] Connection header received
0924 04:13:48.168674 (+ 1097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:48.168677 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:48.168733 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:48.168817 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:48.170375 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.171194 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.172433 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.172677 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.172771 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:48.175615 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:13:48.175619 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:13:48.175621 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:13:48.175651 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:13:48.176993 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:48.177545 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:48.177548 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:48.177550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:48.177642 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:48.177936 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:48.177939 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:48.177940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:48.178117 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:13:48.178192 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:48.178674 (+ 482us) server_negotiation.cc:299] Negotiation successful
0924 04:13:48.178793 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 04:13:48.179682 2217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:48.166959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39493 (local address 127.0.98.1:55187)
0924 04:13:48.167425 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:48.167473 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:48.168417 (+ 944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:48.169048 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:48.169058 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:48.169500 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:13:48.170188 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:48.170209 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.171334 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.171338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:48.172316 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:48.172326 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.172828 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.172833 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:48.173031 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:48.173589 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:13:48.173608 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:13:48.175437 (+ 1829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:13:48.177095 (+ 1658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:48.177100 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:48.177114 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:48.177414 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:48.177730 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:48.177732 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:48.177734 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:48.177859 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:48.178214 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:13:48.178218 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:13:48.178443 (+ 225us) client_negotiation.cc:769] Sending connection context
0924 04:13:48.178591 (+ 148us) client_negotiation.cc:240] Negotiation successful
0924 04:13:48.178831 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":104,"threads_started":1}
I20260924 04:13:48.180913 2324 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39493
I20260924 04:13:48.181046 2324 heartbeater.cc:464] Registering TS with master...
I20260924 04:13:48.181561 2324 heartbeater.cc:511] Master 127.0.98.62:39493 requested a full tablet report, sending...
I20260924 04:13:48.182879 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 2193
I20260924 04:13:48.182979 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:13:48.183061 2120 ts_manager.cc:194] Registered new tserver with Master: fcdc2242e68749518a954101b9abc0f0 (127.0.98.1:38113)
I20260924 04:13:48.183311 392 external_mini_cluster.cc:1428] Setting key 4d2e9e76f2469726b3f66df2de368ee9
I20260924 04:13:48.184460 2120 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55187
I20260924 04:13:48.189708 2330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:48.182889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39493 (local address 127.0.98.1:58695)
0924 04:13:48.183174 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:48.183188 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:48.183380 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:48.183625 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:48.183628 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:48.183818 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:13:48.183948 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:48.183956 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.184866 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.184877 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:48.185606 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:48.185615 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.186012 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.186015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:48.186101 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:48.186554 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:13:48.186567 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:13:48.186942 (+ 375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:13:48.188533 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:48.188535 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:48.188537 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:48.188704 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:48.188937 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:48.188939 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:48.188940 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:48.189003 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:48.189345 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:13:48.189347 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:13:48.189414 (+ 67us) client_negotiation.cc:769] Sending connection context
0924 04:13:48.189483 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 04:13:48.189575 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":75,"threads_started":1}
I20260924 04:13:48.189801 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:48.182984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58695 (local address 127.0.98.62:39493)
0924 04:13:48.183108 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:48.183112 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:48.183266 (+ 154us) server_negotiation.cc:407] Connection header received
0924 04:13:48.183479 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:48.183482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:48.183527 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:48.183600 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:48.184068 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.184755 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.185711 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.185919 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.186000 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:48.187069 (+ 1069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:13:48.187073 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:13:48.187075 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:13:48.187099 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:13:48.188443 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:48.188785 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:48.188788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:48.188790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:48.188865 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:48.189077 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:48.189080 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:48.189081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:48.189270 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:13:48.189364 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:48.189549 (+ 185us) server_negotiation.cc:299] Negotiation successful
0924 04:13:48.189656 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:13:48.210171 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39493
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:13:48.319293 2333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:13:48.319569 2333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:13:48.319615 2333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:13:48.323623 2333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:13:48.323683 2333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:13:48.323722 2333 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 04:13:48.323735 2333 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 04:13:48.323766 2333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:13:48.329435 2333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:13:48.330569 2333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:13:48.331637 2333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:13:48.341049 2339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:13:48.341049 2338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:13:48.341235 2333 server_base.cc:1034] running on GCE node
W20260924 04:13:48.341076 2341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:13:48.341763 2333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:13:48.342403 2333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:13:48.343564 2333 hybrid_clock.cc:648] HybridClock initialized: now 1790223228343540 us; error 36 us; skew 500 ppm
Sep 24 04:13:48 dist-test-slave-p401 krb5kdc[2076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223228, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:13:48.346248 2333 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:13:48.347117 2333 webserver.cc:460] Webserver started at http://127.0.98.2:42031/ using document root <none> and password file <none>
I20260924 04:13:48.347682 2333 fs_manager.cc:359] Metadata directory not provided
I20260924 04:13:48.347748 2333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:13:48.347908 2333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:13:48.349880 2333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/ts-1/data/instance:
uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4"
format_stamp: "Formatted at 2026-09-24 04:13:48 on dist-test-slave-p401"
server_key: "e35dd8fff798f7f83e6bd5f13929a34a"
server_key_iv: "e4eb4bb434adc7cf29c91f10b3a82e0f"
server_key_version: "encryptionkey@0"
I20260924 04:13:48.350370 2333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4"
format_stamp: "Formatted at 2026-09-24 04:13:48 on dist-test-slave-p401"
server_key: "e35dd8fff798f7f83e6bd5f13929a34a"
server_key_iv: "e4eb4bb434adc7cf29c91f10b3a82e0f"
server_key_version: "encryptionkey@0"
I20260924 04:13:48.353960 2333 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260924 04:13:48.356444 2348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:13:48.357638 2333 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:13:48.357735 2333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/ts-1/wal
uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4"
format_stamp: "Formatted at 2026-09-24 04:13:48 on dist-test-slave-p401"
server_key: "e35dd8fff798f7f83e6bd5f13929a34a"
server_key_iv: "e4eb4bb434adc7cf29c91f10b3a82e0f"
server_key_version: "encryptionkey@0"
I20260924 04:13:48.357834 2333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:13:48.380178 2333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:13:48.383194 2333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:13:48.383399 2333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:13:48.384929 2333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:13:48.384982 2333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:13:48.385026 2333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:13:48.385054 2333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 04:13:48 dist-test-slave-p401 krb5kdc[2076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223228, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:13:48.395120 2333 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44715
I20260924 04:13:48.395181 2463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44715 every 8 connection(s)
I20260924 04:13:48.396246 2333 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:13:48.398392 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:48.385709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43455 (local address 127.0.98.62:39493)
0924 04:13:48.385824 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:48.385828 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:48.386942 (+ 1114us) server_negotiation.cc:407] Connection header received
0924 04:13:48.388230 (+ 1288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:48.388234 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:48.388288 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:48.388364 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:48.389992 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.390523 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.391876 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.392099 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.392187 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:48.394698 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:13:48.394702 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:13:48.394704 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:13:48.394730 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:13:48.396422 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:48.397043 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:48.397047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:48.397049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:48.397146 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:48.397459 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:48.397461 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:48.397463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:48.397642 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:13:48.397714 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:48.398171 (+ 457us) server_negotiation.cc:299] Negotiation successful
0924 04:13:48.398268 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 04:13:48.399271 2357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:48.386302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39493 (local address 127.0.98.2:43455)
0924 04:13:48.386784 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:48.386827 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:48.388000 (+ 1173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:48.388591 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:48.388601 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:48.389093 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:13:48.389820 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:48.389840 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.390645 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.390648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:48.391762 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:48.391772 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.392248 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.392252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:48.392419 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:48.392958 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:13:48.392973 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:13:48.394517 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:13:48.396557 (+ 2040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:48.396562 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:48.396575 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:48.396928 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:48.397235 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:48.397237 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:48.397238 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:48.397376 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:48.397714 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:13:48.397719 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:13:48.397978 (+ 259us) client_negotiation.cc:769] Sending connection context
0924 04:13:48.398134 (+ 156us) client_negotiation.cc:240] Negotiation successful
0924 04:13:48.398401 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":137,"threads_started":1}
I20260924 04:13:48.400472 2464 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39493
I20260924 04:13:48.400614 2464 heartbeater.cc:464] Registering TS with master...
I20260924 04:13:48.401157 2464 heartbeater.cc:511] Master 127.0.98.62:39493 requested a full tablet report, sending...
I20260924 04:13:48.402276 2121 ts_manager.cc:194] Registered new tserver with Master: ccbfb74b9f27441ab6aaa9d2e9737ba4 (127.0.98.2:44715)
I20260924 04:13:48.403023 2121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43455
I20260924 04:13:48.406379 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 2333
I20260924 04:13:48.406495 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:13:48.406790 392 external_mini_cluster.cc:1428] Setting key c977f2d5ddb2ddd21441ffdb13038960
I20260924 04:13:48.409807 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:48.402719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46131 (local address 127.0.98.62:39493)
0924 04:13:48.402842 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:48.402847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:48.402987 (+ 140us) server_negotiation.cc:407] Connection header received
0924 04:13:48.403111 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:48.403113 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:48.403149 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:48.403207 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:48.403690 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.404194 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.405213 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.405422 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.405499 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:48.406722 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:13:48.406725 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:13:48.406727 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:13:48.406751 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:13:48.408287 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:48.408733 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:48.408735 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:48.408736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:48.408806 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:48.409045 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:48.409047 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:48.409048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:48.409241 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:13:48.409309 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:48.409550 (+ 241us) server_negotiation.cc:299] Negotiation successful
0924 04:13:48.409644 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 04:13:48.409832 2470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:48.402657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39493 (local address 127.0.98.2:46131)
0924 04:13:48.402917 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:48.402933 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:48.403032 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:48.403272 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:48.403275 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:48.403471 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:13:48.403576 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:48.403583 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.404322 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.404326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:48.405105 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:48.405114 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.405542 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.405545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:48.405619 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:48.406131 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:13:48.406148 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:13:48.406587 (+ 439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:13:48.408397 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:48.408400 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:48.408402 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:48.408644 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:48.408876 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:48.408878 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:48.408881 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:48.408971 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:48.409310 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:13:48.409313 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:13:48.409414 (+ 101us) client_negotiation.cc:769] Sending connection context
0924 04:13:48.409511 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 04:13:48.409648 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":85,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:13:48.431331 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39493
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:13:48.538359 2473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:13:48.538622 2473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:13:48.538688 2473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:13:48.542799 2473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:13:48.542863 2473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:13:48.542922 2473 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 04:13:48.542949 2473 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 04:13:48.542999 2473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:13:48.548224 2473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:13:48.549225 2473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:13:48.550263 2473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:13:48.559458 2479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:13:48.559514 2473 server_base.cc:1034] running on GCE node
W20260924 04:13:48.559469 2481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:13:48.559473 2478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:13:48.560060 2473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:13:48.560684 2473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:13:48.561830 2473 hybrid_clock.cc:648] HybridClock initialized: now 1790223228561808 us; error 34 us; skew 500 ppm
Sep 24 04:13:48 dist-test-slave-p401 krb5kdc[2076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223228, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:13:48.564479 2473 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:13:48.565358 2473 webserver.cc:460] Webserver started at http://127.0.98.3:39789/ using document root <none> and password file <none>
I20260924 04:13:48.565909 2473 fs_manager.cc:359] Metadata directory not provided
I20260924 04:13:48.565969 2473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:13:48.566152 2473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:13:48.568216 2473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/ts-2/data/instance:
uuid: "e3be62bcb7e74350a2dc9655b68aa289"
format_stamp: "Formatted at 2026-09-24 04:13:48 on dist-test-slave-p401"
server_key: "dabe6f4c4ed94696da70c5d13e778b30"
server_key_iv: "1cec7f3607f0ec3ea472bb9afd9cd923"
server_key_version: "encryptionkey@0"
I20260924 04:13:48.568742 2473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e3be62bcb7e74350a2dc9655b68aa289"
format_stamp: "Formatted at 2026-09-24 04:13:48 on dist-test-slave-p401"
server_key: "dabe6f4c4ed94696da70c5d13e778b30"
server_key_iv: "1cec7f3607f0ec3ea472bb9afd9cd923"
server_key_version: "encryptionkey@0"
I20260924 04:13:48.572443 2473 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:13:48.574782 2488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:13:48.576018 2473 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:13:48.576128 2473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/ts-2/wal
uuid: "e3be62bcb7e74350a2dc9655b68aa289"
format_stamp: "Formatted at 2026-09-24 04:13:48 on dist-test-slave-p401"
server_key: "dabe6f4c4ed94696da70c5d13e778b30"
server_key_iv: "1cec7f3607f0ec3ea472bb9afd9cd923"
server_key_version: "encryptionkey@0"
I20260924 04:13:48.576228 2473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:13:48.588778 2473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:13:48.591874 2473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:13:48.592042 2473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:13:48.593448 2473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:13:48.593497 2473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:13:48.593535 2473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:13:48.593562 2473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 04:13:48 dist-test-slave-p401 krb5kdc[2076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223228, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:13:48.603368 2473 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34169
I20260924 04:13:48.603459 2603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34169 every 8 connection(s)
I20260924 04:13:48.604240 2473 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:13:48.607034 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 2473
I20260924 04:13:48.607141 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:13:48.607321 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:48.594250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48765 (local address 127.0.98.62:39493)
0924 04:13:48.594359 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:48.594363 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:48.595450 (+ 1087us) server_negotiation.cc:407] Connection header received
0924 04:13:48.596440 (+ 990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:48.596444 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:48.596503 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:48.596561 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:48.598239 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.599051 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.600815 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.601033 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.601120 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:48.603984 (+ 2864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:13:48.603988 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:13:48.603991 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:13:48.604018 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:13:48.605490 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:48.605975 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:48.605979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:48.605980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:48.606060 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:48.606300 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:48.606302 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:48.606303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:48.606473 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:13:48.606537 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:48.607009 (+ 472us) server_negotiation.cc:299] Negotiation successful
0924 04:13:48.607134 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 04:13:48.607553 392 external_mini_cluster.cc:1428] Setting key f094456664f36cbcf05aeffb145da11a
I20260924 04:13:48.608096 2497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:48.594817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39493 (local address 127.0.98.3:48765)
0924 04:13:48.595311 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:48.595354 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:48.596190 (+ 836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:48.597013 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:48.597023 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:48.597374 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:13:48.598061 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:48.598081 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.599187 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.599191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:48.600651 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:48.600662 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.601190 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.601194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:48.601410 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:48.602059 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:13:48.602082 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:13:48.603810 (+ 1728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:13:48.605588 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:48.605592 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:48.605602 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:48.605883 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:48.606127 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:48.606129 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:48.606130 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:48.606245 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:48.606537 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:13:48.606541 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:13:48.606755 (+ 214us) client_negotiation.cc:769] Sending connection context
0924 04:13:48.606909 (+ 154us) client_negotiation.cc:240] Negotiation successful
0924 04:13:48.607148 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":130,"threads_started":1}
I20260924 04:13:48.609261 2604 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39493
I20260924 04:13:48.609408 2604 heartbeater.cc:464] Registering TS with master...
I20260924 04:13:48.609966 2604 heartbeater.cc:511] Master 127.0.98.62:39493 requested a full tablet report, sending...
I20260924 04:13:48.610920 2120 ts_manager.cc:194] Registered new tserver with Master: e3be62bcb7e74350a2dc9655b68aa289 (127.0.98.3:34169)
I20260924 04:13:48.611545 2120 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48765
I20260924 04:13:48.617075 2610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:48.611180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39493 (local address 127.0.98.3:59873)
0924 04:13:48.611509 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:48.611520 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:48.611594 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:48.611798 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:48.611801 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:48.612011 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:13:48.612128 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:48.612135 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.613056 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.613058 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:48.613600 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:48.613606 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.613977 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.613979 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:48.614038 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:48.614430 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:13:48.614440 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:13:48.614706 (+ 266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:13:48.616047 (+ 1341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:48.616049 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:48.616050 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:48.616214 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:48.616400 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:48.616402 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:48.616403 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:48.616463 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:48.616715 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:13:48.616717 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:13:48.616782 (+ 65us) client_negotiation.cc:769] Sending connection context
0924 04:13:48.616848 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 04:13:48.616953 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":100,"threads_started":1}
I20260924 04:13:48.617090 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:48.611274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59873 (local address 127.0.98.62:39493)
0924 04:13:48.611373 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:48.611377 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:48.611582 (+ 205us) server_negotiation.cc:407] Connection header received
0924 04:13:48.611651 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:48.611654 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:48.611693 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:48.611764 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:48.612225 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.612953 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.613689 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.613898 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.613971 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:48.614810 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:13:48.614813 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:13:48.614815 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:13:48.614835 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:13:48.615963 (+ 1128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:48.616280 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:48.616282 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:48.616283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:48.616343 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:48.616517 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:48.616519 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:48.616520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:48.616658 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:13:48.616722 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:48.616903 (+ 181us) server_negotiation.cc:299] Negotiation successful
0924 04:13:48.616985 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 04:13:48.620986 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 04:13:48 dist-test-slave-p401 krb5kdc[2076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223228, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 04:13:48 dist-test-slave-p401 krb5kdc[2076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223228, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:13:48.644523 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:48.636909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50536 (local address 127.0.98.62:39493)
0924 04:13:48.637035 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:48.637039 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:48.637051 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:13:48.637092 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:48.637094 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:48.637132 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:48.637205 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:48.637869 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.638490 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.639316 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.639560 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.639645 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:48.641835 (+ 2190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:13:48.641838 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:13:48.641840 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:13:48.641865 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:13:48.643113 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:48.643538 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:48.643542 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:48.643543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:48.643625 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:48.643830 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:48.643833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:48.643834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:48.644005 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:13:48.644107 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:48.644256 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 04:13:48.644349 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 04:13:48.647591 2120 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50536:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:13:48.650450 2120 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:13:48.664160 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:48.658612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41050 (local address 127.0.98.3:34169)
0924 04:13:48.658901 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:48.658912 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:48.658934 (+ 22us) server_negotiation.cc:407] Connection header received
0924 04:13:48.659134 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:48.659140 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:48.659400 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:48.659556 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:13:48.659970 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.661140 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.662819 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.663525 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.663626 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:48.663781 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:48.663905 (+ 124us) server_negotiation.cc:299] Negotiation successful
0924 04:13:48.663998 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":101,"threads_started":1}
I20260924 04:13:48.664150 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:48.658463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34169 (local address 127.0.0.1:41050)
0924 04:13:48.658772 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:48.658830 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:48.658976 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:48.659731 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:48.659737 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:48.659759 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:13:48.659870 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:48.659877 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.661241 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.661245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:48.662700 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:48.662710 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.663639 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.663642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:48.663741 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:48.663774 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 04:13:48.663889 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 04:13:48.664007 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":73,"threads_started":1}
I20260924 04:13:48.664455 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:48.659335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48816 (local address 127.0.98.1:38113)
0924 04:13:48.659684 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:48.659700 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:48.660034 (+ 334us) server_negotiation.cc:407] Connection header received
0924 04:13:48.660147 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:48.660152 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:48.660341 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:48.660493 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:13:48.660683 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.661689 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.662935 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.663599 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.663695 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:48.663843 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:48.664215 (+ 372us) server_negotiation.cc:299] Negotiation successful
0924 04:13:48.664310 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":77,"threads_started":1}
I20260924 04:13:48.664695 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:48.659157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38113 (local address 127.0.0.1:48816)
0924 04:13:48.659947 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:48.659959 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:48.660049 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:48.660471 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:48.660474 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:48.660489 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:13:48.660581 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:48.660586 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.661802 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.661805 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:48.662816 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:48.662826 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.663899 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.663902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:48.663978 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:48.663997 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 04:13:48.664057 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 04:13:48.664108 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":729,"thread_start_us":107,"threads_started":1}
I20260924 04:13:48.665957 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:48.659453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44715 (local address 127.0.0.1:35604)
0924 04:13:48.660215 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:48.660226 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:48.660306 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:48.660704 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:48.660707 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:48.660763 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:13:48.660838 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:48.660843 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.662083 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.662086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:48.663138 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:48.663147 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.664312 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.664316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:48.665653 (+ 1337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:48.665678 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 04:13:48.665750 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 04:13:48.665819 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"thread_start_us":75,"threads_started":1}
I20260924 04:13:48.666338 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:48.659821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35604 (local address 127.0.98.2:44715)
0924 04:13:48.660204 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:48.660214 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:48.660298 (+ 84us) server_negotiation.cc:407] Connection header received
0924 04:13:48.660409 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:48.660413 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:48.660587 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:48.660722 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:13:48.660933 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.661967 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.663286 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.664022 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.664114 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:48.664248 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:48.666117 (+ 1869us) server_negotiation.cc:299] Negotiation successful
0924 04:13:48.666209 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":98,"threads_started":1}
I20260924 04:13:48.667027 2538 tablet_service.cc:1467] Processing CreateTablet for tablet f87d42a9aaef4a978fe75ac5a2d5d40d (DEFAULT_TABLE table=test-table [id=f0f4df16202545409d3268190a955886]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:13:48.667382 2258 tablet_service.cc:1467] Processing CreateTablet for tablet f87d42a9aaef4a978fe75ac5a2d5d40d (DEFAULT_TABLE table=test-table [id=f0f4df16202545409d3268190a955886]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:13:48.668226 2538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f87d42a9aaef4a978fe75ac5a2d5d40d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:13:48.668311 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f87d42a9aaef4a978fe75ac5a2d5d40d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:13:48.669746 2398 tablet_service.cc:1467] Processing CreateTablet for tablet f87d42a9aaef4a978fe75ac5a2d5d40d (DEFAULT_TABLE table=test-table [id=f0f4df16202545409d3268190a955886]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:13:48.670697 2398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f87d42a9aaef4a978fe75ac5a2d5d40d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:13:48.674494 2627 tablet_bootstrap.cc:492] T f87d42a9aaef4a978fe75ac5a2d5d40d P fcdc2242e68749518a954101b9abc0f0: Bootstrap starting.
I20260924 04:13:48.676426 2628 tablet_bootstrap.cc:492] T f87d42a9aaef4a978fe75ac5a2d5d40d P e3be62bcb7e74350a2dc9655b68aa289: Bootstrap starting.
I20260924 04:13:48.676676 2627 tablet_bootstrap.cc:654] T f87d42a9aaef4a978fe75ac5a2d5d40d P fcdc2242e68749518a954101b9abc0f0: Neither blocks nor log segments found. Creating new log.
I20260924 04:13:48.677361 2627 log.cc:824] T f87d42a9aaef4a978fe75ac5a2d5d40d P fcdc2242e68749518a954101b9abc0f0: Log is configured to *not* fsync() on all Append() calls
I20260924 04:13:48.677814 2629 tablet_bootstrap.cc:492] T f87d42a9aaef4a978fe75ac5a2d5d40d P ccbfb74b9f27441ab6aaa9d2e9737ba4: Bootstrap starting.
I20260924 04:13:48.678572 2628 tablet_bootstrap.cc:654] T f87d42a9aaef4a978fe75ac5a2d5d40d P e3be62bcb7e74350a2dc9655b68aa289: Neither blocks nor log segments found. Creating new log.
I20260924 04:13:48.679425 2628 log.cc:824] T f87d42a9aaef4a978fe75ac5a2d5d40d P e3be62bcb7e74350a2dc9655b68aa289: Log is configured to *not* fsync() on all Append() calls
I20260924 04:13:48.679600 2627 tablet_bootstrap.cc:492] T f87d42a9aaef4a978fe75ac5a2d5d40d P fcdc2242e68749518a954101b9abc0f0: No bootstrap required, opened a new log
I20260924 04:13:48.679689 2629 tablet_bootstrap.cc:654] T f87d42a9aaef4a978fe75ac5a2d5d40d P ccbfb74b9f27441ab6aaa9d2e9737ba4: Neither blocks nor log segments found. Creating new log.
I20260924 04:13:48.679783 2627 ts_tablet_manager.cc:1397] T f87d42a9aaef4a978fe75ac5a2d5d40d P fcdc2242e68749518a954101b9abc0f0: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 04:13:48.680430 2629 log.cc:824] T f87d42a9aaef4a978fe75ac5a2d5d40d P ccbfb74b9f27441ab6aaa9d2e9737ba4: Log is configured to *not* fsync() on all Append() calls
I20260924 04:13:48.681219 2628 tablet_bootstrap.cc:492] T f87d42a9aaef4a978fe75ac5a2d5d40d P e3be62bcb7e74350a2dc9655b68aa289: No bootstrap required, opened a new log
I20260924 04:13:48.681394 2628 ts_tablet_manager.cc:1397] T f87d42a9aaef4a978fe75ac5a2d5d40d P e3be62bcb7e74350a2dc9655b68aa289: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 04:13:48.682427 2629 tablet_bootstrap.cc:492] T f87d42a9aaef4a978fe75ac5a2d5d40d P ccbfb74b9f27441ab6aaa9d2e9737ba4: No bootstrap required, opened a new log
I20260924 04:13:48.682550 2629 ts_tablet_manager.cc:1397] T f87d42a9aaef4a978fe75ac5a2d5d40d P ccbfb74b9f27441ab6aaa9d2e9737ba4: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 04:13:48.683588 2627 raft_consensus.cc:348] T f87d42a9aaef4a978fe75ac5a2d5d40d P fcdc2242e68749518a954101b9abc0f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3be62bcb7e74350a2dc9655b68aa289" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34169 } } peers { permanent_uuid: "fcdc2242e68749518a954101b9abc0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38113 } } peers { permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 } }
I20260924 04:13:48.683832 2627 raft_consensus.cc:374] T f87d42a9aaef4a978fe75ac5a2d5d40d P fcdc2242e68749518a954101b9abc0f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:13:48.683933 2627 raft_consensus.cc:729] T f87d42a9aaef4a978fe75ac5a2d5d40d P fcdc2242e68749518a954101b9abc0f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcdc2242e68749518a954101b9abc0f0, State: Initialized, Role: FOLLOWER
I20260924 04:13:48.684367 2627 consensus_queue.cc:260] T f87d42a9aaef4a978fe75ac5a2d5d40d P fcdc2242e68749518a954101b9abc0f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3be62bcb7e74350a2dc9655b68aa289" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34169 } } peers { permanent_uuid: "fcdc2242e68749518a954101b9abc0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38113 } } peers { permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 } }
I20260924 04:13:48.685027 2324 heartbeater.cc:503] Master 127.0.98.62:39493 was elected leader, sending a full tablet report...
I20260924 04:13:48.685060 2628 raft_consensus.cc:348] T f87d42a9aaef4a978fe75ac5a2d5d40d P e3be62bcb7e74350a2dc9655b68aa289 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3be62bcb7e74350a2dc9655b68aa289" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34169 } } peers { permanent_uuid: "fcdc2242e68749518a954101b9abc0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38113 } } peers { permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 } }
I20260924 04:13:48.685281 2628 raft_consensus.cc:374] T f87d42a9aaef4a978fe75ac5a2d5d40d P e3be62bcb7e74350a2dc9655b68aa289 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:13:48.685333 2628 raft_consensus.cc:729] T f87d42a9aaef4a978fe75ac5a2d5d40d P e3be62bcb7e74350a2dc9655b68aa289 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3be62bcb7e74350a2dc9655b68aa289, State: Initialized, Role: FOLLOWER
I20260924 04:13:48.685496 2627 ts_tablet_manager.cc:1428] T f87d42a9aaef4a978fe75ac5a2d5d40d P fcdc2242e68749518a954101b9abc0f0: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 04:13:48.685500 2629 raft_consensus.cc:348] T f87d42a9aaef4a978fe75ac5a2d5d40d P ccbfb74b9f27441ab6aaa9d2e9737ba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3be62bcb7e74350a2dc9655b68aa289" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34169 } } peers { permanent_uuid: "fcdc2242e68749518a954101b9abc0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38113 } } peers { permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 } }
I20260924 04:13:48.685732 2629 raft_consensus.cc:374] T f87d42a9aaef4a978fe75ac5a2d5d40d P ccbfb74b9f27441ab6aaa9d2e9737ba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:13:48.685786 2629 raft_consensus.cc:729] T f87d42a9aaef4a978fe75ac5a2d5d40d P ccbfb74b9f27441ab6aaa9d2e9737ba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccbfb74b9f27441ab6aaa9d2e9737ba4, State: Initialized, Role: FOLLOWER
I20260924 04:13:48.685823 2628 consensus_queue.cc:260] T f87d42a9aaef4a978fe75ac5a2d5d40d P e3be62bcb7e74350a2dc9655b68aa289 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3be62bcb7e74350a2dc9655b68aa289" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34169 } } peers { permanent_uuid: "fcdc2242e68749518a954101b9abc0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38113 } } peers { permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 } }
I20260924 04:13:48.686201 2629 consensus_queue.cc:260] T f87d42a9aaef4a978fe75ac5a2d5d40d P ccbfb74b9f27441ab6aaa9d2e9737ba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3be62bcb7e74350a2dc9655b68aa289" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34169 } } peers { permanent_uuid: "fcdc2242e68749518a954101b9abc0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38113 } } peers { permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 } }
I20260924 04:13:48.686723 2628 ts_tablet_manager.cc:1428] T f87d42a9aaef4a978fe75ac5a2d5d40d P e3be62bcb7e74350a2dc9655b68aa289: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 04:13:48.686753 2464 heartbeater.cc:503] Master 127.0.98.62:39493 was elected leader, sending a full tablet report...
I20260924 04:13:48.687028 2604 heartbeater.cc:503] Master 127.0.98.62:39493 was elected leader, sending a full tablet report...
I20260924 04:13:48.687079 2629 ts_tablet_manager.cc:1428] T f87d42a9aaef4a978fe75ac5a2d5d40d P ccbfb74b9f27441ab6aaa9d2e9737ba4: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260924 04:13:48.820020 2633 raft_consensus.cc:482] T f87d42a9aaef4a978fe75ac5a2d5d40d P fcdc2242e68749518a954101b9abc0f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:13:48.820231 2633 raft_consensus.cc:504] T f87d42a9aaef4a978fe75ac5a2d5d40d P fcdc2242e68749518a954101b9abc0f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3be62bcb7e74350a2dc9655b68aa289" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34169 } } peers { permanent_uuid: "fcdc2242e68749518a954101b9abc0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38113 } } peers { permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 } }
I20260924 04:13:48.821300 2633 leader_election.cc:288] T f87d42a9aaef4a978fe75ac5a2d5d40d P fcdc2242e68749518a954101b9abc0f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3be62bcb7e74350a2dc9655b68aa289 (127.0.98.3:34169), ccbfb74b9f27441ab6aaa9d2e9737ba4 (127.0.98.2:44715)
I20260924 04:13:48.826155 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:48.822087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35403 (local address 127.0.98.2:44715)
0924 04:13:48.822200 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:48.822204 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:48.822292 (+ 88us) server_negotiation.cc:407] Connection header received
0924 04:13:48.822381 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:48.822383 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:48.822421 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:48.822496 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:13:48.822697 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.823434 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.824802 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.825402 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.825501 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:48.825571 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:48.825965 (+ 394us) server_negotiation.cc:299] Negotiation successful
0924 04:13:48.826031 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 04:13:48.826155 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:48.821672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41631 (local address 127.0.98.3:34169)
0924 04:13:48.821785 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:48.821789 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:48.821896 (+ 107us) server_negotiation.cc:407] Connection header received
0924 04:13:48.822048 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:48.822051 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:48.822103 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:48.822193 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:13:48.822476 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.823285 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.824805 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.825398 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.825501 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:48.825563 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:48.825965 (+ 402us) server_negotiation.cc:299] Negotiation successful
0924 04:13:48.826031 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 04:13:48.826272 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:48.821954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44715 (local address 127.0.98.1:35403)
0924 04:13:48.822214 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:48.822227 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:48.822304 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:48.822512 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:48.822514 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:48.822528 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:13:48.822610 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:48.822614 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.823519 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.823522 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:48.824706 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:48.824712 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.825560 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.825565 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:48.825813 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:48.825863 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 04:13:48.825952 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 04:13:48.826068 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":146,"thread_start_us":68,"threads_started":1}
I20260924 04:13:48.826275 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:48.821544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34169 (local address 127.0.98.1:41631)
0924 04:13:48.821815 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:48.821829 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:48.821938 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:48.822231 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:48.822234 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:48.822255 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:13:48.822379 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:48.822386 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.823383 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.823386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:48.824700 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:48.824712 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.825560 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.825565 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:48.825821 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:48.825859 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 04:13:48.825952 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 04:13:48.826062 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":88,"thread_start_us":101,"threads_started":1}
I20260924 04:13:48.826917 2558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f87d42a9aaef4a978fe75ac5a2d5d40d" candidate_uuid: "fcdc2242e68749518a954101b9abc0f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3be62bcb7e74350a2dc9655b68aa289" is_pre_election: true
I20260924 04:13:48.826910 2418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f87d42a9aaef4a978fe75ac5a2d5d40d" candidate_uuid: "fcdc2242e68749518a954101b9abc0f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" is_pre_election: true
I20260924 04:13:48.827179 2558 raft_consensus.cc:2454] T f87d42a9aaef4a978fe75ac5a2d5d40d P e3be62bcb7e74350a2dc9655b68aa289 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fcdc2242e68749518a954101b9abc0f0 in term 0.
I20260924 04:13:48.827180 2418 raft_consensus.cc:2454] T f87d42a9aaef4a978fe75ac5a2d5d40d P ccbfb74b9f27441ab6aaa9d2e9737ba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fcdc2242e68749518a954101b9abc0f0 in term 0.
I20260924 04:13:48.827668 2209 leader_election.cc:302] T f87d42a9aaef4a978fe75ac5a2d5d40d P fcdc2242e68749518a954101b9abc0f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e3be62bcb7e74350a2dc9655b68aa289, fcdc2242e68749518a954101b9abc0f0; no voters:
I20260924 04:13:48.827929 2633 raft_consensus.cc:2781] T f87d42a9aaef4a978fe75ac5a2d5d40d P fcdc2242e68749518a954101b9abc0f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:13:48.828001 2633 raft_consensus.cc:482] T f87d42a9aaef4a978fe75ac5a2d5d40d P fcdc2242e68749518a954101b9abc0f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:13:48.828030 2633 raft_consensus.cc:3037] T f87d42a9aaef4a978fe75ac5a2d5d40d P fcdc2242e68749518a954101b9abc0f0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:13:48.829098 2633 raft_consensus.cc:504] T f87d42a9aaef4a978fe75ac5a2d5d40d P fcdc2242e68749518a954101b9abc0f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3be62bcb7e74350a2dc9655b68aa289" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34169 } } peers { permanent_uuid: "fcdc2242e68749518a954101b9abc0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38113 } } peers { permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 } }
I20260924 04:13:48.829393 2633 leader_election.cc:288] T f87d42a9aaef4a978fe75ac5a2d5d40d P fcdc2242e68749518a954101b9abc0f0 [CANDIDATE]: Term 1 election: Requested vote from peers e3be62bcb7e74350a2dc9655b68aa289 (127.0.98.3:34169), ccbfb74b9f27441ab6aaa9d2e9737ba4 (127.0.98.2:44715)
I20260924 04:13:48.829687 2558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f87d42a9aaef4a978fe75ac5a2d5d40d" candidate_uuid: "fcdc2242e68749518a954101b9abc0f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3be62bcb7e74350a2dc9655b68aa289"
I20260924 04:13:48.829741 2418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f87d42a9aaef4a978fe75ac5a2d5d40d" candidate_uuid: "fcdc2242e68749518a954101b9abc0f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4"
I20260924 04:13:48.829797 2558 raft_consensus.cc:3037] T f87d42a9aaef4a978fe75ac5a2d5d40d P e3be62bcb7e74350a2dc9655b68aa289 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:13:48.829850 2418 raft_consensus.cc:3037] T f87d42a9aaef4a978fe75ac5a2d5d40d P ccbfb74b9f27441ab6aaa9d2e9737ba4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:13:48.830770 2558 raft_consensus.cc:2454] T f87d42a9aaef4a978fe75ac5a2d5d40d P e3be62bcb7e74350a2dc9655b68aa289 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcdc2242e68749518a954101b9abc0f0 in term 1.
I20260924 04:13:48.830857 2418 raft_consensus.cc:2454] T f87d42a9aaef4a978fe75ac5a2d5d40d P ccbfb74b9f27441ab6aaa9d2e9737ba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcdc2242e68749518a954101b9abc0f0 in term 1.
I20260924 04:13:48.831038 2209 leader_election.cc:302] T f87d42a9aaef4a978fe75ac5a2d5d40d P fcdc2242e68749518a954101b9abc0f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e3be62bcb7e74350a2dc9655b68aa289, fcdc2242e68749518a954101b9abc0f0; no voters:
I20260924 04:13:48.831180 2633 raft_consensus.cc:2781] T f87d42a9aaef4a978fe75ac5a2d5d40d P fcdc2242e68749518a954101b9abc0f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:13:48.831398 2633 raft_consensus.cc:686] T f87d42a9aaef4a978fe75ac5a2d5d40d P fcdc2242e68749518a954101b9abc0f0 [term 1 LEADER]: Becoming Leader. State: Replica: fcdc2242e68749518a954101b9abc0f0, State: Running, Role: LEADER
I20260924 04:13:48.831689 2633 consensus_queue.cc:237] T f87d42a9aaef4a978fe75ac5a2d5d40d P fcdc2242e68749518a954101b9abc0f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3be62bcb7e74350a2dc9655b68aa289" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34169 } } peers { permanent_uuid: "fcdc2242e68749518a954101b9abc0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38113 } } peers { permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 } }
I20260924 04:13:48.834422 2121 catalog_manager.cc:5495] T f87d42a9aaef4a978fe75ac5a2d5d40d P fcdc2242e68749518a954101b9abc0f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to fcdc2242e68749518a954101b9abc0f0 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "fcdc2242e68749518a954101b9abc0f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3be62bcb7e74350a2dc9655b68aa289" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fcdc2242e68749518a954101b9abc0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 } health_report { overall_health: UNKNOWN } } }
W20260924 04:13:48.855405 2605 tablet.cc:2369] T f87d42a9aaef4a978fe75ac5a2d5d40d P e3be62bcb7e74350a2dc9655b68aa289: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 04:13:48 dist-test-slave-p401 krb5kdc[2076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223227, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:13:48.888311 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:48.880931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39493 (local address 127.0.0.1:50540)
0924 04:13:48.881210 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:48.881221 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:48.881328 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:48.881540 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:48.881542 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:48.881954 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:13:48.882048 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:48.882056 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.882767 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.882769 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:48.883388 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:48.883396 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.883744 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.883747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:48.883849 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:48.884353 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:13:48.884363 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:13:48.885645 (+ 1282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:13:48.887077 (+ 1432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:48.887083 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:48.887087 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:48.887325 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:48.887519 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:48.887521 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:48.887523 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:48.887590 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:48.887853 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:13:48.887858 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:13:48.887959 (+ 101us) client_negotiation.cc:769] Sending connection context
0924 04:13:48.888039 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 04:13:48.888146 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":5,"thread_start_us":89,"threads_started":1}
I20260924 04:13:48.888322 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:48.881007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50540 (local address 127.0.98.62:39493)
0924 04:13:48.881123 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:48.881128 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:48.881287 (+ 159us) server_negotiation.cc:407] Connection header received
0924 04:13:48.881414 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:48.881417 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:48.881452 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:48.881515 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:48.882175 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.882683 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.883469 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.883666 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.883735 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:48.885753 (+ 2018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:13:48.885756 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:13:48.885758 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:13:48.885778 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:13:48.887003 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:48.887388 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:48.887390 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:48.887391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:48.887456 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:48.887643 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:48.887645 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:48.887646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:48.887789 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:13:48.887850 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:48.888072 (+ 222us) server_negotiation.cc:299] Negotiation successful
0924 04:13:48.888195 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":36}
I20260924 04:13:48.891651 2121 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50540:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 04:13:48.892287 2121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:13:48.896580 2538 tablet_service.cc:1467] Processing CreateTablet for tablet 8762101914044dd78139e1c385c794fe (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=18bf2eb364c548ec8af6c7915929ce51]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 04:13:48.896544 2258 tablet_service.cc:1467] Processing CreateTablet for tablet 8762101914044dd78139e1c385c794fe (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=18bf2eb364c548ec8af6c7915929ce51]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 04:13:48.896559 2398 tablet_service.cc:1467] Processing CreateTablet for tablet 8762101914044dd78139e1c385c794fe (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=18bf2eb364c548ec8af6c7915929ce51]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 04:13:48.897233 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8762101914044dd78139e1c385c794fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:13:48.897233 2398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8762101914044dd78139e1c385c794fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:13:48.897236 2538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8762101914044dd78139e1c385c794fe. 1 dirs total, 0 dirs full, 0 dirs failed
W20260924 04:13:48.897755 2465 tablet.cc:2369] T f87d42a9aaef4a978fe75ac5a2d5d40d P ccbfb74b9f27441ab6aaa9d2e9737ba4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:13:48.899878 2628 tablet_bootstrap.cc:492] T 8762101914044dd78139e1c385c794fe P e3be62bcb7e74350a2dc9655b68aa289: Bootstrap starting.
I20260924 04:13:48.900081 2627 tablet_bootstrap.cc:492] T 8762101914044dd78139e1c385c794fe P fcdc2242e68749518a954101b9abc0f0: Bootstrap starting.
I20260924 04:13:48.900779 2628 tablet_bootstrap.cc:654] T 8762101914044dd78139e1c385c794fe P e3be62bcb7e74350a2dc9655b68aa289: Neither blocks nor log segments found. Creating new log.
I20260924 04:13:48.901149 2627 tablet_bootstrap.cc:654] T 8762101914044dd78139e1c385c794fe P fcdc2242e68749518a954101b9abc0f0: Neither blocks nor log segments found. Creating new log.
I20260924 04:13:48.901281 2629 tablet_bootstrap.cc:492] T 8762101914044dd78139e1c385c794fe P ccbfb74b9f27441ab6aaa9d2e9737ba4: Bootstrap starting.
I20260924 04:13:48.902132 2629 tablet_bootstrap.cc:654] T 8762101914044dd78139e1c385c794fe P ccbfb74b9f27441ab6aaa9d2e9737ba4: Neither blocks nor log segments found. Creating new log.
I20260924 04:13:48.902197 2627 tablet_bootstrap.cc:492] T 8762101914044dd78139e1c385c794fe P fcdc2242e68749518a954101b9abc0f0: No bootstrap required, opened a new log
I20260924 04:13:48.902275 2627 ts_tablet_manager.cc:1397] T 8762101914044dd78139e1c385c794fe P fcdc2242e68749518a954101b9abc0f0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 04:13:48.902706 2627 raft_consensus.cc:348] T 8762101914044dd78139e1c385c794fe P fcdc2242e68749518a954101b9abc0f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcdc2242e68749518a954101b9abc0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38113 } } peers { permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 } } peers { permanent_uuid: "e3be62bcb7e74350a2dc9655b68aa289" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34169 } }
I20260924 04:13:48.902817 2627 raft_consensus.cc:374] T 8762101914044dd78139e1c385c794fe P fcdc2242e68749518a954101b9abc0f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:13:48.902849 2627 raft_consensus.cc:729] T 8762101914044dd78139e1c385c794fe P fcdc2242e68749518a954101b9abc0f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcdc2242e68749518a954101b9abc0f0, State: Initialized, Role: FOLLOWER
I20260924 04:13:48.902961 2627 consensus_queue.cc:260] T 8762101914044dd78139e1c385c794fe P fcdc2242e68749518a954101b9abc0f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcdc2242e68749518a954101b9abc0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38113 } } peers { permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 } } peers { permanent_uuid: "e3be62bcb7e74350a2dc9655b68aa289" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34169 } }
I20260924 04:13:48.903208 2628 tablet_bootstrap.cc:492] T 8762101914044dd78139e1c385c794fe P e3be62bcb7e74350a2dc9655b68aa289: No bootstrap required, opened a new log
I20260924 04:13:48.903216 2627 ts_tablet_manager.cc:1428] T 8762101914044dd78139e1c385c794fe P fcdc2242e68749518a954101b9abc0f0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:13:48.903182 2633 tablet_replica.cc:439] T 8762101914044dd78139e1c385c794fe P fcdc2242e68749518a954101b9abc0f0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcdc2242e68749518a954101b9abc0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38113 } } peers { permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 } } peers { permanent_uuid: "e3be62bcb7e74350a2dc9655b68aa289" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34169 } } }
I20260924 04:13:48.903311 2628 ts_tablet_manager.cc:1397] T 8762101914044dd78139e1c385c794fe P e3be62bcb7e74350a2dc9655b68aa289: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 04:13:48.903318 2633 tablet_replica.cc:442] T 8762101914044dd78139e1c385c794fe P fcdc2242e68749518a954101b9abc0f0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:13:48.903208 2629 tablet_bootstrap.cc:492] T 8762101914044dd78139e1c385c794fe P ccbfb74b9f27441ab6aaa9d2e9737ba4: No bootstrap required, opened a new log
I20260924 04:13:48.903705 2629 ts_tablet_manager.cc:1397] T 8762101914044dd78139e1c385c794fe P ccbfb74b9f27441ab6aaa9d2e9737ba4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:13:48.903728 2628 raft_consensus.cc:348] T 8762101914044dd78139e1c385c794fe P e3be62bcb7e74350a2dc9655b68aa289 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcdc2242e68749518a954101b9abc0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38113 } } peers { permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 } } peers { permanent_uuid: "e3be62bcb7e74350a2dc9655b68aa289" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34169 } }
I20260924 04:13:48.903818 2628 raft_consensus.cc:374] T 8762101914044dd78139e1c385c794fe P e3be62bcb7e74350a2dc9655b68aa289 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:13:48.903848 2628 raft_consensus.cc:729] T 8762101914044dd78139e1c385c794fe P e3be62bcb7e74350a2dc9655b68aa289 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3be62bcb7e74350a2dc9655b68aa289, State: Initialized, Role: FOLLOWER
I20260924 04:13:48.903960 2628 consensus_queue.cc:260] T 8762101914044dd78139e1c385c794fe P e3be62bcb7e74350a2dc9655b68aa289 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcdc2242e68749518a954101b9abc0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38113 } } peers { permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 } } peers { permanent_uuid: "e3be62bcb7e74350a2dc9655b68aa289" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34169 } }
I20260924 04:13:48.904127 2629 raft_consensus.cc:348] T 8762101914044dd78139e1c385c794fe P ccbfb74b9f27441ab6aaa9d2e9737ba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcdc2242e68749518a954101b9abc0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38113 } } peers { permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 } } peers { permanent_uuid: "e3be62bcb7e74350a2dc9655b68aa289" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34169 } }
I20260924 04:13:48.904196 2628 ts_tablet_manager.cc:1428] T 8762101914044dd78139e1c385c794fe P e3be62bcb7e74350a2dc9655b68aa289: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 04:13:48.904220 2629 raft_consensus.cc:374] T 8762101914044dd78139e1c385c794fe P ccbfb74b9f27441ab6aaa9d2e9737ba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:13:48.904247 2629 raft_consensus.cc:729] T 8762101914044dd78139e1c385c794fe P ccbfb74b9f27441ab6aaa9d2e9737ba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccbfb74b9f27441ab6aaa9d2e9737ba4, State: Initialized, Role: FOLLOWER
I20260924 04:13:48.904220 2634 tablet_replica.cc:439] T 8762101914044dd78139e1c385c794fe P e3be62bcb7e74350a2dc9655b68aa289: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcdc2242e68749518a954101b9abc0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38113 } } peers { permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 } } peers { permanent_uuid: "e3be62bcb7e74350a2dc9655b68aa289" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34169 } } }
I20260924 04:13:48.904353 2634 tablet_replica.cc:442] T 8762101914044dd78139e1c385c794fe P e3be62bcb7e74350a2dc9655b68aa289: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:13:48.904356 2629 consensus_queue.cc:260] T 8762101914044dd78139e1c385c794fe P ccbfb74b9f27441ab6aaa9d2e9737ba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcdc2242e68749518a954101b9abc0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38113 } } peers { permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 } } peers { permanent_uuid: "e3be62bcb7e74350a2dc9655b68aa289" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34169 } }
I20260924 04:13:48.904567 2635 tablet_replica.cc:439] T 8762101914044dd78139e1c385c794fe P ccbfb74b9f27441ab6aaa9d2e9737ba4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcdc2242e68749518a954101b9abc0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38113 } } peers { permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 } } peers { permanent_uuid: "e3be62bcb7e74350a2dc9655b68aa289" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34169 } } }
I20260924 04:13:48.904693 2635 tablet_replica.cc:442] T 8762101914044dd78139e1c385c794fe P ccbfb74b9f27441ab6aaa9d2e9737ba4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:13:48.904580 2629 ts_tablet_manager.cc:1428] T 8762101914044dd78139e1c385c794fe P ccbfb74b9f27441ab6aaa9d2e9737ba4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 04:13:48.906244 2634 raft_consensus.cc:482] T 8762101914044dd78139e1c385c794fe P e3be62bcb7e74350a2dc9655b68aa289 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:13:48.906333 2634 raft_consensus.cc:504] T 8762101914044dd78139e1c385c794fe P e3be62bcb7e74350a2dc9655b68aa289 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcdc2242e68749518a954101b9abc0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38113 } } peers { permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 } } peers { permanent_uuid: "e3be62bcb7e74350a2dc9655b68aa289" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34169 } }
I20260924 04:13:48.907342 2634 leader_election.cc:288] T 8762101914044dd78139e1c385c794fe P e3be62bcb7e74350a2dc9655b68aa289 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fcdc2242e68749518a954101b9abc0f0 (127.0.98.1:38113), ccbfb74b9f27441ab6aaa9d2e9737ba4 (127.0.98.2:44715)
I20260924 04:13:48.914777 2497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:48.907476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38113 (local address 127.0.98.3:58701)
0924 04:13:48.907580 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:48.907593 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:48.907699 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:48.912040 (+ 4341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:48.912043 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:48.912059 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:13:48.912161 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:48.912168 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.912961 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.912964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:48.913703 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:48.913711 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.914404 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.914407 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:48.914505 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:48.914529 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 04:13:48.914594 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 04:13:48.914659 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 04:13:48.914788 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:48.911674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58701 (local address 127.0.98.1:38113)
0924 04:13:48.911773 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:48.911777 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:48.911789 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:13:48.911826 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:48.911829 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:48.911868 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:48.911940 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:13:48.912273 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.912856 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.913806 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.914301 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.914396 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:48.914452 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:48.914615 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 04:13:48.914680 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 04:13:48.915174 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:48.907790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47695 (local address 127.0.98.2:44715)
0924 04:13:48.911346 (+ 3556us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:48.911350 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:48.911361 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:13:48.911396 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:48.911398 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:48.911442 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:48.911500 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:13:48.911815 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.912574 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.914047 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.914568 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.914657 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:48.914707 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:48.915000 (+ 293us) server_negotiation.cc:299] Negotiation successful
0924 04:13:48.915072 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3490}
I20260924 04:13:48.915262 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8762101914044dd78139e1c385c794fe" candidate_uuid: "e3be62bcb7e74350a2dc9655b68aa289" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcdc2242e68749518a954101b9abc0f0" is_pre_election: true
I20260924 04:13:48.915460 2278 raft_consensus.cc:2454] T 8762101914044dd78139e1c385c794fe P fcdc2242e68749518a954101b9abc0f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3be62bcb7e74350a2dc9655b68aa289 in term 0.
I20260924 04:13:48.915545 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:48.907663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44715 (local address 127.0.98.3:47695)
0924 04:13:48.907923 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:48.907933 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:48.907992 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:48.911539 (+ 3547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:48.911542 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:48.911562 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:13:48.911687 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:48.911694 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.912703 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.912706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:48.913933 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:48.913943 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.914776 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.914780 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:48.914868 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:48.914890 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 04:13:48.915318 (+ 428us) client_negotiation.cc:240] Negotiation successful
0924 04:13:48.915414 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":126,"threads_started":1}
I20260924 04:13:48.915892 2418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8762101914044dd78139e1c385c794fe" candidate_uuid: "e3be62bcb7e74350a2dc9655b68aa289" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" is_pre_election: true
I20260924 04:13:48.916008 2418 raft_consensus.cc:2454] T 8762101914044dd78139e1c385c794fe P ccbfb74b9f27441ab6aaa9d2e9737ba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3be62bcb7e74350a2dc9655b68aa289 in term 0.
I20260924 04:13:48.916016 2489 leader_election.cc:302] T 8762101914044dd78139e1c385c794fe P e3be62bcb7e74350a2dc9655b68aa289 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e3be62bcb7e74350a2dc9655b68aa289, fcdc2242e68749518a954101b9abc0f0; no voters:
I20260924 04:13:48.916447 2634 raft_consensus.cc:2781] T 8762101914044dd78139e1c385c794fe P e3be62bcb7e74350a2dc9655b68aa289 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:13:48.916499 2634 raft_consensus.cc:482] T 8762101914044dd78139e1c385c794fe P e3be62bcb7e74350a2dc9655b68aa289 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:13:48.916525 2634 raft_consensus.cc:3037] T 8762101914044dd78139e1c385c794fe P e3be62bcb7e74350a2dc9655b68aa289 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:13:48.917373 2634 raft_consensus.cc:504] T 8762101914044dd78139e1c385c794fe P e3be62bcb7e74350a2dc9655b68aa289 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcdc2242e68749518a954101b9abc0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38113 } } peers { permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 } } peers { permanent_uuid: "e3be62bcb7e74350a2dc9655b68aa289" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34169 } }
I20260924 04:13:48.917656 2634 leader_election.cc:288] T 8762101914044dd78139e1c385c794fe P e3be62bcb7e74350a2dc9655b68aa289 [CANDIDATE]: Term 1 election: Requested vote from peers fcdc2242e68749518a954101b9abc0f0 (127.0.98.1:38113), ccbfb74b9f27441ab6aaa9d2e9737ba4 (127.0.98.2:44715)
I20260924 04:13:48.917979 2418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8762101914044dd78139e1c385c794fe" candidate_uuid: "e3be62bcb7e74350a2dc9655b68aa289" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4"
I20260924 04:13:48.918073 2418 raft_consensus.cc:3037] T 8762101914044dd78139e1c385c794fe P ccbfb74b9f27441ab6aaa9d2e9737ba4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:13:48.918843 2418 raft_consensus.cc:2454] T 8762101914044dd78139e1c385c794fe P ccbfb74b9f27441ab6aaa9d2e9737ba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3be62bcb7e74350a2dc9655b68aa289 in term 1.
I20260924 04:13:48.919051 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8762101914044dd78139e1c385c794fe" candidate_uuid: "e3be62bcb7e74350a2dc9655b68aa289" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcdc2242e68749518a954101b9abc0f0"
I20260924 04:13:48.919081 2489 leader_election.cc:302] T 8762101914044dd78139e1c385c794fe P e3be62bcb7e74350a2dc9655b68aa289 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ccbfb74b9f27441ab6aaa9d2e9737ba4, e3be62bcb7e74350a2dc9655b68aa289; no voters:
I20260924 04:13:48.919153 2278 raft_consensus.cc:3037] T 8762101914044dd78139e1c385c794fe P fcdc2242e68749518a954101b9abc0f0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:13:48.919333 2634 raft_consensus.cc:2781] T 8762101914044dd78139e1c385c794fe P e3be62bcb7e74350a2dc9655b68aa289 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:13:48.919520 2634 raft_consensus.cc:686] T 8762101914044dd78139e1c385c794fe P e3be62bcb7e74350a2dc9655b68aa289 [term 1 LEADER]: Becoming Leader. State: Replica: e3be62bcb7e74350a2dc9655b68aa289, State: Running, Role: LEADER
I20260924 04:13:48.919827 2634 consensus_queue.cc:237] T 8762101914044dd78139e1c385c794fe P e3be62bcb7e74350a2dc9655b68aa289 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcdc2242e68749518a954101b9abc0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38113 } } peers { permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 } } peers { permanent_uuid: "e3be62bcb7e74350a2dc9655b68aa289" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34169 } }
I20260924 04:13:48.920186 2278 raft_consensus.cc:2454] T 8762101914044dd78139e1c385c794fe P fcdc2242e68749518a954101b9abc0f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3be62bcb7e74350a2dc9655b68aa289 in term 1.
I20260924 04:13:48.921993 2650 tablet_replica.cc:439] T 8762101914044dd78139e1c385c794fe P e3be62bcb7e74350a2dc9655b68aa289: TxnStatusTablet state changed. Reason: New leader e3be62bcb7e74350a2dc9655b68aa289. Latest consensus state: current_term: 1 leader_uuid: "e3be62bcb7e74350a2dc9655b68aa289" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcdc2242e68749518a954101b9abc0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38113 } } peers { permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 } } peers { permanent_uuid: "e3be62bcb7e74350a2dc9655b68aa289" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34169 } } }
I20260924 04:13:48.922081 2650 tablet_replica.cc:442] T 8762101914044dd78139e1c385c794fe P e3be62bcb7e74350a2dc9655b68aa289: This TxnStatusTablet replica's current role is: LEADER
I20260924 04:13:48.922318 2652 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 04:13:48.922493 2652 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 04:13:48.923033 2119 catalog_manager.cc:5495] T 8762101914044dd78139e1c385c794fe P e3be62bcb7e74350a2dc9655b68aa289 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3be62bcb7e74350a2dc9655b68aa289 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "e3be62bcb7e74350a2dc9655b68aa289" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcdc2242e68749518a954101b9abc0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3be62bcb7e74350a2dc9655b68aa289" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34169 } health_report { overall_health: HEALTHY } } }
W20260924 04:13:48.928148 2325 tablet.cc:2369] T 8762101914044dd78139e1c385c794fe P fcdc2242e68749518a954101b9abc0f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:13:48.935349 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:48.931194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34169 (local address 127.0.0.1:41054)
0924 04:13:48.931466 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:48.931480 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:48.931588 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:48.931919 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:48.931922 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:48.931929 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:13:48.932025 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:48.932030 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.933009 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.933011 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:48.933634 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:48.933643 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.934215 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.934219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:48.934314 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:48.934359 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:13:48.935057 (+ 698us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:13:48.935070 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 04:13:48.935130 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 04:13:48.935199 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157}
I20260924 04:13:48.935369 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:48.931299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41054 (local address 127.0.98.3:34169)
0924 04:13:48.931433 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:48.931438 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:48.931544 (+ 106us) server_negotiation.cc:407] Connection header received
0924 04:13:48.931713 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:48.931716 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:48.931775 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:48.931884 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:13:48.932127 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.932889 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.933758 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.934093 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.934205 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:48.934485 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:13:48.934944 (+ 459us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:13:48.935049 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:48.935142 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 04:13:48.935194 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 04:13:48.939864 2278 raft_consensus.cc:1261] T 8762101914044dd78139e1c385c794fe P fcdc2242e68749518a954101b9abc0f0 [term 1 FOLLOWER]: Refusing update from remote peer e3be62bcb7e74350a2dc9655b68aa289: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:13:48.939972 2418 raft_consensus.cc:1261] T 8762101914044dd78139e1c385c794fe P ccbfb74b9f27441ab6aaa9d2e9737ba4 [term 1 FOLLOWER]: Refusing update from remote peer e3be62bcb7e74350a2dc9655b68aa289: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:13:48.940393 2634 consensus_queue.cc:1035] T 8762101914044dd78139e1c385c794fe P e3be62bcb7e74350a2dc9655b68aa289 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcdc2242e68749518a954101b9abc0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:13:48.940559 2650 consensus_queue.cc:1035] T 8762101914044dd78139e1c385c794fe P e3be62bcb7e74350a2dc9655b68aa289 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:13:48.943575 2633 tablet_replica.cc:439] T 8762101914044dd78139e1c385c794fe P fcdc2242e68749518a954101b9abc0f0: TxnStatusTablet state changed. Reason: New leader e3be62bcb7e74350a2dc9655b68aa289. Latest consensus state: current_term: 1 leader_uuid: "e3be62bcb7e74350a2dc9655b68aa289" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcdc2242e68749518a954101b9abc0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38113 } } peers { permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 } } peers { permanent_uuid: "e3be62bcb7e74350a2dc9655b68aa289" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34169 } } }
I20260924 04:13:48.943683 2633 tablet_replica.cc:442] T 8762101914044dd78139e1c385c794fe P fcdc2242e68749518a954101b9abc0f0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:13:48.943921 2635 tablet_replica.cc:439] T 8762101914044dd78139e1c385c794fe P ccbfb74b9f27441ab6aaa9d2e9737ba4: TxnStatusTablet state changed. Reason: New leader e3be62bcb7e74350a2dc9655b68aa289. Latest consensus state: current_term: 1 leader_uuid: "e3be62bcb7e74350a2dc9655b68aa289" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcdc2242e68749518a954101b9abc0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38113 } } peers { permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 } } peers { permanent_uuid: "e3be62bcb7e74350a2dc9655b68aa289" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34169 } } }
I20260924 04:13:48.944016 2635 tablet_replica.cc:442] T 8762101914044dd78139e1c385c794fe P ccbfb74b9f27441ab6aaa9d2e9737ba4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:13:48.946983 2634 tablet_replica.cc:439] T 8762101914044dd78139e1c385c794fe P e3be62bcb7e74350a2dc9655b68aa289: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e3be62bcb7e74350a2dc9655b68aa289" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcdc2242e68749518a954101b9abc0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38113 } } peers { permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 } } peers { permanent_uuid: "e3be62bcb7e74350a2dc9655b68aa289" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34169 } } }
I20260924 04:13:48.947088 2634 tablet_replica.cc:442] T 8762101914044dd78139e1c385c794fe P e3be62bcb7e74350a2dc9655b68aa289: This TxnStatusTablet replica's current role is: LEADER
I20260924 04:13:48.947324 2655 mvcc.cc:205] Tried to move back new op lower bound from 7332754345731301376 to 7332754345658089472. Current Snapshot: MvccSnapshot[applied={T|T < 7332754345731301376}]
I20260924 04:13:48.947523 2635 tablet_replica.cc:439] T 8762101914044dd78139e1c385c794fe P ccbfb74b9f27441ab6aaa9d2e9737ba4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e3be62bcb7e74350a2dc9655b68aa289" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcdc2242e68749518a954101b9abc0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38113 } } peers { permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 } } peers { permanent_uuid: "e3be62bcb7e74350a2dc9655b68aa289" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34169 } } }
I20260924 04:13:48.947620 2635 tablet_replica.cc:442] T 8762101914044dd78139e1c385c794fe P ccbfb74b9f27441ab6aaa9d2e9737ba4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:13:48.947988 2656 mvcc.cc:205] Tried to move back new op lower bound from 7332754345731301376 to 7332754345658089472. Current Snapshot: MvccSnapshot[applied={T|T < 7332754345731301376}]
I20260924 04:13:48.951289 2634 tablet_replica.cc:439] T 8762101914044dd78139e1c385c794fe P e3be62bcb7e74350a2dc9655b68aa289: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e3be62bcb7e74350a2dc9655b68aa289" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcdc2242e68749518a954101b9abc0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38113 } } peers { permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 } } peers { permanent_uuid: "e3be62bcb7e74350a2dc9655b68aa289" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34169 } } }
I20260924 04:13:48.951391 2634 tablet_replica.cc:442] T 8762101914044dd78139e1c385c794fe P e3be62bcb7e74350a2dc9655b68aa289: This TxnStatusTablet replica's current role is: LEADER
I20260924 04:13:48.952953 2633 tablet_replica.cc:439] T 8762101914044dd78139e1c385c794fe P fcdc2242e68749518a954101b9abc0f0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e3be62bcb7e74350a2dc9655b68aa289" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcdc2242e68749518a954101b9abc0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38113 } } peers { permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 } } peers { permanent_uuid: "e3be62bcb7e74350a2dc9655b68aa289" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34169 } } }
I20260924 04:13:48.953066 2633 tablet_replica.cc:442] T 8762101914044dd78139e1c385c794fe P fcdc2242e68749518a954101b9abc0f0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:13:48.954746 2657 mvcc.cc:205] Tried to move back new op lower bound from 7332754345731301376 to 7332754345658089472. Current Snapshot: MvccSnapshot[applied={T|T < 7332754345731301376 or (T in {7332754345731301376})}]
I20260924 04:13:48.958184 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:48.954623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48818 (local address 127.0.98.1:38113)
0924 04:13:48.954717 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:48.954721 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:48.954871 (+ 150us) server_negotiation.cc:407] Connection header received
0924 04:13:48.955029 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:48.955032 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:48.955086 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:48.955187 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:13:48.955442 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.955945 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.956866 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.957090 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.957184 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:48.957431 (+ 247us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:13:48.957839 (+ 408us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:13:48.957938 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:48.958010 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 04:13:48.958063 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 04:13:48.968582 2666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:48.965166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34169 (local address 127.0.98.1:50631)
0924 04:13:48.965450 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:48.965463 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:48.965545 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:48.965755 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:48.965756 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:48.965762 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:13:48.965873 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:48.965878 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.966781 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.966783 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:48.967431 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:48.967439 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.967952 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.967954 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:48.968015 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:48.968052 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:13:48.968331 (+ 279us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:13:48.968336 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 04:13:48.968393 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 04:13:48.968454 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":85,"threads_started":1}
I20260924 04:13:48.968581 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:48.965267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50631 (local address 127.0.98.3:34169)
0924 04:13:48.965383 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:48.965387 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:48.965524 (+ 137us) server_negotiation.cc:407] Connection header received
0924 04:13:48.965624 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:48.965627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:48.965677 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:48.965752 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:13:48.965954 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.966703 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.967529 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.967858 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.967960 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:48.968176 (+ 216us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:13:48.968255 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:13:48.968318 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:48.968407 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 04:13:48.968454 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:13:48.975441 2418 raft_consensus.cc:1261] T f87d42a9aaef4a978fe75ac5a2d5d40d P ccbfb74b9f27441ab6aaa9d2e9737ba4 [term 1 FOLLOWER]: Refusing update from remote peer fcdc2242e68749518a954101b9abc0f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:13:48.975622 2558 raft_consensus.cc:1261] T f87d42a9aaef4a978fe75ac5a2d5d40d P e3be62bcb7e74350a2dc9655b68aa289 [term 1 FOLLOWER]: Refusing update from remote peer fcdc2242e68749518a954101b9abc0f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:13:48.975847 2638 consensus_queue.cc:1035] T f87d42a9aaef4a978fe75ac5a2d5d40d P fcdc2242e68749518a954101b9abc0f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:13:48.976073 2633 consensus_queue.cc:1035] T f87d42a9aaef4a978fe75ac5a2d5d40d P fcdc2242e68749518a954101b9abc0f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3be62bcb7e74350a2dc9655b68aa289" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:13:48.997022 2610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:48.993585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38113 (local address 127.0.98.3:40925)
0924 04:13:48.993801 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:48.993813 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:48.993887 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:48.994077 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:48.994078 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:48.994083 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:13:48.994180 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:48.994184 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.995140 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.995144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:48.995932 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:48.995941 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.996465 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.996468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:48.996533 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:48.996572 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:13:48.996813 (+ 241us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:13:48.996817 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 04:13:48.996855 (+ 38us) client_negotiation.cc:240] Negotiation successful
0924 04:13:48.996918 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260924 04:13:48.997043 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:48.993667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40925 (local address 127.0.98.1:38113)
0924 04:13:48.993774 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:48.993778 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:48.993849 (+ 71us) server_negotiation.cc:407] Connection header received
0924 04:13:48.993951 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:48.993954 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:48.993999 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:48.994075 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:13:48.994256 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.995034 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.996037 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:48.996373 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:48.996476 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:48.996671 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:13:48.996746 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:13:48.996809 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:48.996866 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 04:13:48.996922 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 04:13:49.024174 2119 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:13:49.024389 2119 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:13:49.027087 2119 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2fac17aff8b64408b4db746cefeeddbc: Sending DeleteTablet for 3 replicas of tablet f87d42a9aaef4a978fe75ac5a2d5d40d
I20260924 04:13:49.028110 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet f87d42a9aaef4a978fe75ac5a2d5d40d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:13:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48816
I20260924 04:13:49.028790 2398 tablet_service.cc:1514] Processing DeleteTablet for tablet f87d42a9aaef4a978fe75ac5a2d5d40d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:13:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35604
I20260924 04:13:49.028777 2119 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50536:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:13:49.029182 2119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:13:49.029464 2538 tablet_service.cc:1514] Processing DeleteTablet for tablet f87d42a9aaef4a978fe75ac5a2d5d40d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:13:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41050
I20260924 04:13:49.030035 2674 tablet_replica.cc:330] T f87d42a9aaef4a978fe75ac5a2d5d40d P fcdc2242e68749518a954101b9abc0f0: stopping tablet replica
I20260924 04:13:49.030354 2674 raft_consensus.cc:2229] T f87d42a9aaef4a978fe75ac5a2d5d40d P fcdc2242e68749518a954101b9abc0f0 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:13:49.030733 2674 raft_consensus.cc:2258] T f87d42a9aaef4a978fe75ac5a2d5d40d P fcdc2242e68749518a954101b9abc0f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:13:49.032018 2675 tablet_replica.cc:330] T f87d42a9aaef4a978fe75ac5a2d5d40d P ccbfb74b9f27441ab6aaa9d2e9737ba4: stopping tablet replica
I20260924 04:13:49.032367 2675 raft_consensus.cc:2229] T f87d42a9aaef4a978fe75ac5a2d5d40d P ccbfb74b9f27441ab6aaa9d2e9737ba4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:13:49.032612 2675 raft_consensus.cc:2258] T f87d42a9aaef4a978fe75ac5a2d5d40d P ccbfb74b9f27441ab6aaa9d2e9737ba4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:13:49.033262 2676 tablet_replica.cc:330] T f87d42a9aaef4a978fe75ac5a2d5d40d P e3be62bcb7e74350a2dc9655b68aa289: stopping tablet replica
I20260924 04:13:49.033525 2676 raft_consensus.cc:2229] T f87d42a9aaef4a978fe75ac5a2d5d40d P e3be62bcb7e74350a2dc9655b68aa289 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:13:49.033675 2676 raft_consensus.cc:2258] T f87d42a9aaef4a978fe75ac5a2d5d40d P e3be62bcb7e74350a2dc9655b68aa289 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:13:49.034222 2674 ts_tablet_manager.cc:1905] T f87d42a9aaef4a978fe75ac5a2d5d40d P fcdc2242e68749518a954101b9abc0f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:13:49.035457 2398 tablet_service.cc:1467] Processing CreateTablet for tablet 85dde4779eea43d6a486cd8df9c05c33 (DEFAULT_TABLE table=test-table [id=cc07be245c614358ae98eda01c2dd4a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:13:49.035712 2398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85dde4779eea43d6a486cd8df9c05c33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:13:49.037352 2676 ts_tablet_manager.cc:1905] T f87d42a9aaef4a978fe75ac5a2d5d40d P e3be62bcb7e74350a2dc9655b68aa289: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:13:49.037992 2629 tablet_bootstrap.cc:492] T 85dde4779eea43d6a486cd8df9c05c33 P ccbfb74b9f27441ab6aaa9d2e9737ba4: Bootstrap starting.
I20260924 04:13:49.038226 2538 tablet_service.cc:1467] Processing CreateTablet for tablet 85dde4779eea43d6a486cd8df9c05c33 (DEFAULT_TABLE table=test-table [id=cc07be245c614358ae98eda01c2dd4a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:13:49.038455 2538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85dde4779eea43d6a486cd8df9c05c33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:13:49.038959 2629 tablet_bootstrap.cc:654] T 85dde4779eea43d6a486cd8df9c05c33 P ccbfb74b9f27441ab6aaa9d2e9737ba4: Neither blocks nor log segments found. Creating new log.
I20260924 04:13:49.038973 2258 tablet_service.cc:1467] Processing CreateTablet for tablet 85dde4779eea43d6a486cd8df9c05c33 (DEFAULT_TABLE table=test-table [id=cc07be245c614358ae98eda01c2dd4a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:13:49.039207 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85dde4779eea43d6a486cd8df9c05c33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:13:49.041353 2627 tablet_bootstrap.cc:492] T 85dde4779eea43d6a486cd8df9c05c33 P fcdc2242e68749518a954101b9abc0f0: Bootstrap starting.
I20260924 04:13:49.042203 2627 tablet_bootstrap.cc:654] T 85dde4779eea43d6a486cd8df9c05c33 P fcdc2242e68749518a954101b9abc0f0: Neither blocks nor log segments found. Creating new log.
I20260924 04:13:49.043006 2674 ts_tablet_manager.cc:1918] T f87d42a9aaef4a978fe75ac5a2d5d40d P fcdc2242e68749518a954101b9abc0f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:13:49.043100 2674 log.cc:1196] T f87d42a9aaef4a978fe75ac5a2d5d40d P fcdc2242e68749518a954101b9abc0f0: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/ts-0/wal/wals/f87d42a9aaef4a978fe75ac5a2d5d40d
I20260924 04:13:49.043428 2674 ts_tablet_manager.cc:1939] T f87d42a9aaef4a978fe75ac5a2d5d40d P fcdc2242e68749518a954101b9abc0f0: Deleting consensus metadata
I20260924 04:13:49.043905 2675 ts_tablet_manager.cc:1905] T f87d42a9aaef4a978fe75ac5a2d5d40d P ccbfb74b9f27441ab6aaa9d2e9737ba4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:13:49.044548 2105 catalog_manager.cc:4841] TS fcdc2242e68749518a954101b9abc0f0 (127.0.98.1:38113): tablet f87d42a9aaef4a978fe75ac5a2d5d40d (table test-table [id=f0f4df16202545409d3268190a955886]) successfully deleted
I20260924 04:13:49.046736 2628 tablet_bootstrap.cc:492] T 85dde4779eea43d6a486cd8df9c05c33 P e3be62bcb7e74350a2dc9655b68aa289: Bootstrap starting.
I20260924 04:13:49.047703 2628 tablet_bootstrap.cc:654] T 85dde4779eea43d6a486cd8df9c05c33 P e3be62bcb7e74350a2dc9655b68aa289: Neither blocks nor log segments found. Creating new log.
I20260924 04:13:49.048548 2676 ts_tablet_manager.cc:1918] T f87d42a9aaef4a978fe75ac5a2d5d40d P e3be62bcb7e74350a2dc9655b68aa289: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:13:49.048640 2676 log.cc:1196] T f87d42a9aaef4a978fe75ac5a2d5d40d P e3be62bcb7e74350a2dc9655b68aa289: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/ts-2/wal/wals/f87d42a9aaef4a978fe75ac5a2d5d40d
I20260924 04:13:49.048583 2675 ts_tablet_manager.cc:1918] T f87d42a9aaef4a978fe75ac5a2d5d40d P ccbfb74b9f27441ab6aaa9d2e9737ba4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:13:49.048803 2675 log.cc:1196] T f87d42a9aaef4a978fe75ac5a2d5d40d P ccbfb74b9f27441ab6aaa9d2e9737ba4: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/ts-1/wal/wals/f87d42a9aaef4a978fe75ac5a2d5d40d
I20260924 04:13:49.048947 2676 ts_tablet_manager.cc:1939] T f87d42a9aaef4a978fe75ac5a2d5d40d P e3be62bcb7e74350a2dc9655b68aa289: Deleting consensus metadata
I20260924 04:13:49.049187 2675 ts_tablet_manager.cc:1939] T f87d42a9aaef4a978fe75ac5a2d5d40d P ccbfb74b9f27441ab6aaa9d2e9737ba4: Deleting consensus metadata
I20260924 04:13:49.050079 2105 catalog_manager.cc:4841] TS e3be62bcb7e74350a2dc9655b68aa289 (127.0.98.3:34169): tablet f87d42a9aaef4a978fe75ac5a2d5d40d (table test-table [id=f0f4df16202545409d3268190a955886]) successfully deleted
I20260924 04:13:49.050230 2105 catalog_manager.cc:4841] TS ccbfb74b9f27441ab6aaa9d2e9737ba4 (127.0.98.2:44715): tablet f87d42a9aaef4a978fe75ac5a2d5d40d (table test-table [id=f0f4df16202545409d3268190a955886]) successfully deleted
I20260924 04:13:49.051041 2629 tablet_bootstrap.cc:492] T 85dde4779eea43d6a486cd8df9c05c33 P ccbfb74b9f27441ab6aaa9d2e9737ba4: No bootstrap required, opened a new log
I20260924 04:13:49.051126 2629 ts_tablet_manager.cc:1397] T 85dde4779eea43d6a486cd8df9c05c33 P ccbfb74b9f27441ab6aaa9d2e9737ba4: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.001s
I20260924 04:13:49.051600 2629 raft_consensus.cc:348] T 85dde4779eea43d6a486cd8df9c05c33 P ccbfb74b9f27441ab6aaa9d2e9737ba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcdc2242e68749518a954101b9abc0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38113 } } peers { permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 } } peers { permanent_uuid: "e3be62bcb7e74350a2dc9655b68aa289" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34169 } }
I20260924 04:13:49.051975 2627 tablet_bootstrap.cc:492] T 85dde4779eea43d6a486cd8df9c05c33 P fcdc2242e68749518a954101b9abc0f0: No bootstrap required, opened a new log
I20260924 04:13:49.052058 2627 ts_tablet_manager.cc:1397] T 85dde4779eea43d6a486cd8df9c05c33 P fcdc2242e68749518a954101b9abc0f0: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.002s
I20260924 04:13:49.052477 2627 raft_consensus.cc:348] T 85dde4779eea43d6a486cd8df9c05c33 P fcdc2242e68749518a954101b9abc0f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcdc2242e68749518a954101b9abc0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38113 } } peers { permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 } } peers { permanent_uuid: "e3be62bcb7e74350a2dc9655b68aa289" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34169 } }
I20260924 04:13:49.052584 2627 raft_consensus.cc:374] T 85dde4779eea43d6a486cd8df9c05c33 P fcdc2242e68749518a954101b9abc0f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:13:49.052615 2627 raft_consensus.cc:729] T 85dde4779eea43d6a486cd8df9c05c33 P fcdc2242e68749518a954101b9abc0f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcdc2242e68749518a954101b9abc0f0, State: Initialized, Role: FOLLOWER
I20260924 04:13:49.052737 2627 consensus_queue.cc:260] T 85dde4779eea43d6a486cd8df9c05c33 P fcdc2242e68749518a954101b9abc0f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcdc2242e68749518a954101b9abc0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38113 } } peers { permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 } } peers { permanent_uuid: "e3be62bcb7e74350a2dc9655b68aa289" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34169 } }
I20260924 04:13:49.052757 2629 raft_consensus.cc:374] T 85dde4779eea43d6a486cd8df9c05c33 P ccbfb74b9f27441ab6aaa9d2e9737ba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:13:49.052835 2629 raft_consensus.cc:729] T 85dde4779eea43d6a486cd8df9c05c33 P ccbfb74b9f27441ab6aaa9d2e9737ba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccbfb74b9f27441ab6aaa9d2e9737ba4, State: Initialized, Role: FOLLOWER
I20260924 04:13:49.052954 2629 consensus_queue.cc:260] T 85dde4779eea43d6a486cd8df9c05c33 P ccbfb74b9f27441ab6aaa9d2e9737ba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcdc2242e68749518a954101b9abc0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38113 } } peers { permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 } } peers { permanent_uuid: "e3be62bcb7e74350a2dc9655b68aa289" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34169 } }
I20260924 04:13:49.053017 2627 ts_tablet_manager.cc:1428] T 85dde4779eea43d6a486cd8df9c05c33 P fcdc2242e68749518a954101b9abc0f0: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 04:13:49.053231 2629 ts_tablet_manager.cc:1428] T 85dde4779eea43d6a486cd8df9c05c33 P ccbfb74b9f27441ab6aaa9d2e9737ba4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:13:49.055122 2628 tablet_bootstrap.cc:492] T 85dde4779eea43d6a486cd8df9c05c33 P e3be62bcb7e74350a2dc9655b68aa289: No bootstrap required, opened a new log
I20260924 04:13:49.055202 2628 ts_tablet_manager.cc:1397] T 85dde4779eea43d6a486cd8df9c05c33 P e3be62bcb7e74350a2dc9655b68aa289: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20260924 04:13:49.055634 2628 raft_consensus.cc:348] T 85dde4779eea43d6a486cd8df9c05c33 P e3be62bcb7e74350a2dc9655b68aa289 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcdc2242e68749518a954101b9abc0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38113 } } peers { permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 } } peers { permanent_uuid: "e3be62bcb7e74350a2dc9655b68aa289" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34169 } }
I20260924 04:13:49.055737 2628 raft_consensus.cc:374] T 85dde4779eea43d6a486cd8df9c05c33 P e3be62bcb7e74350a2dc9655b68aa289 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:13:49.055766 2628 raft_consensus.cc:729] T 85dde4779eea43d6a486cd8df9c05c33 P e3be62bcb7e74350a2dc9655b68aa289 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3be62bcb7e74350a2dc9655b68aa289, State: Initialized, Role: FOLLOWER
I20260924 04:13:49.055883 2628 consensus_queue.cc:260] T 85dde4779eea43d6a486cd8df9c05c33 P e3be62bcb7e74350a2dc9655b68aa289 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcdc2242e68749518a954101b9abc0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38113 } } peers { permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 } } peers { permanent_uuid: "e3be62bcb7e74350a2dc9655b68aa289" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34169 } }
I20260924 04:13:49.056133 2628 ts_tablet_manager.cc:1428] T 85dde4779eea43d6a486cd8df9c05c33 P e3be62bcb7e74350a2dc9655b68aa289: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:13:49.344045 2650 raft_consensus.cc:482] T 85dde4779eea43d6a486cd8df9c05c33 P e3be62bcb7e74350a2dc9655b68aa289 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:13:49.344177 2650 raft_consensus.cc:504] T 85dde4779eea43d6a486cd8df9c05c33 P e3be62bcb7e74350a2dc9655b68aa289 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcdc2242e68749518a954101b9abc0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38113 } } peers { permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 } } peers { permanent_uuid: "e3be62bcb7e74350a2dc9655b68aa289" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34169 } }
I20260924 04:13:49.344554 2650 leader_election.cc:288] T 85dde4779eea43d6a486cd8df9c05c33 P e3be62bcb7e74350a2dc9655b68aa289 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fcdc2242e68749518a954101b9abc0f0 (127.0.98.1:38113), ccbfb74b9f27441ab6aaa9d2e9737ba4 (127.0.98.2:44715)
I20260924 04:13:49.344985 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85dde4779eea43d6a486cd8df9c05c33" candidate_uuid: "e3be62bcb7e74350a2dc9655b68aa289" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcdc2242e68749518a954101b9abc0f0" is_pre_election: true
I20260924 04:13:49.345050 2417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85dde4779eea43d6a486cd8df9c05c33" candidate_uuid: "e3be62bcb7e74350a2dc9655b68aa289" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" is_pre_election: true
I20260924 04:13:49.345130 2278 raft_consensus.cc:2454] T 85dde4779eea43d6a486cd8df9c05c33 P fcdc2242e68749518a954101b9abc0f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3be62bcb7e74350a2dc9655b68aa289 in term 0.
I20260924 04:13:49.345189 2417 raft_consensus.cc:2454] T 85dde4779eea43d6a486cd8df9c05c33 P ccbfb74b9f27441ab6aaa9d2e9737ba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3be62bcb7e74350a2dc9655b68aa289 in term 0.
I20260924 04:13:49.345463 2489 leader_election.cc:302] T 85dde4779eea43d6a486cd8df9c05c33 P e3be62bcb7e74350a2dc9655b68aa289 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e3be62bcb7e74350a2dc9655b68aa289, fcdc2242e68749518a954101b9abc0f0; no voters:
I20260924 04:13:49.345654 2650 raft_consensus.cc:2781] T 85dde4779eea43d6a486cd8df9c05c33 P e3be62bcb7e74350a2dc9655b68aa289 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:13:49.345711 2650 raft_consensus.cc:482] T 85dde4779eea43d6a486cd8df9c05c33 P e3be62bcb7e74350a2dc9655b68aa289 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:13:49.345738 2650 raft_consensus.cc:3037] T 85dde4779eea43d6a486cd8df9c05c33 P e3be62bcb7e74350a2dc9655b68aa289 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:13:49.346674 2650 raft_consensus.cc:504] T 85dde4779eea43d6a486cd8df9c05c33 P e3be62bcb7e74350a2dc9655b68aa289 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcdc2242e68749518a954101b9abc0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38113 } } peers { permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 } } peers { permanent_uuid: "e3be62bcb7e74350a2dc9655b68aa289" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34169 } }
I20260924 04:13:49.346966 2650 leader_election.cc:288] T 85dde4779eea43d6a486cd8df9c05c33 P e3be62bcb7e74350a2dc9655b68aa289 [CANDIDATE]: Term 1 election: Requested vote from peers fcdc2242e68749518a954101b9abc0f0 (127.0.98.1:38113), ccbfb74b9f27441ab6aaa9d2e9737ba4 (127.0.98.2:44715)
I20260924 04:13:49.347313 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85dde4779eea43d6a486cd8df9c05c33" candidate_uuid: "e3be62bcb7e74350a2dc9655b68aa289" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcdc2242e68749518a954101b9abc0f0"
I20260924 04:13:49.347352 2417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85dde4779eea43d6a486cd8df9c05c33" candidate_uuid: "e3be62bcb7e74350a2dc9655b68aa289" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4"
I20260924 04:13:49.347430 2278 raft_consensus.cc:3037] T 85dde4779eea43d6a486cd8df9c05c33 P fcdc2242e68749518a954101b9abc0f0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:13:49.347497 2417 raft_consensus.cc:3037] T 85dde4779eea43d6a486cd8df9c05c33 P ccbfb74b9f27441ab6aaa9d2e9737ba4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:13:49.348398 2278 raft_consensus.cc:2454] T 85dde4779eea43d6a486cd8df9c05c33 P fcdc2242e68749518a954101b9abc0f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3be62bcb7e74350a2dc9655b68aa289 in term 1.
I20260924 04:13:49.348400 2417 raft_consensus.cc:2454] T 85dde4779eea43d6a486cd8df9c05c33 P ccbfb74b9f27441ab6aaa9d2e9737ba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3be62bcb7e74350a2dc9655b68aa289 in term 1.
I20260924 04:13:49.348737 2489 leader_election.cc:302] T 85dde4779eea43d6a486cd8df9c05c33 P e3be62bcb7e74350a2dc9655b68aa289 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e3be62bcb7e74350a2dc9655b68aa289, fcdc2242e68749518a954101b9abc0f0; no voters:
I20260924 04:13:49.348873 2650 raft_consensus.cc:2781] T 85dde4779eea43d6a486cd8df9c05c33 P e3be62bcb7e74350a2dc9655b68aa289 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:13:49.348954 2650 raft_consensus.cc:686] T 85dde4779eea43d6a486cd8df9c05c33 P e3be62bcb7e74350a2dc9655b68aa289 [term 1 LEADER]: Becoming Leader. State: Replica: e3be62bcb7e74350a2dc9655b68aa289, State: Running, Role: LEADER
I20260924 04:13:49.349083 2650 consensus_queue.cc:237] T 85dde4779eea43d6a486cd8df9c05c33 P e3be62bcb7e74350a2dc9655b68aa289 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcdc2242e68749518a954101b9abc0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38113 } } peers { permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 } } peers { permanent_uuid: "e3be62bcb7e74350a2dc9655b68aa289" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34169 } }
I20260924 04:13:49.350590 2119 catalog_manager.cc:5495] T 85dde4779eea43d6a486cd8df9c05c33 P e3be62bcb7e74350a2dc9655b68aa289 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3be62bcb7e74350a2dc9655b68aa289 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "e3be62bcb7e74350a2dc9655b68aa289" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcdc2242e68749518a954101b9abc0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3be62bcb7e74350a2dc9655b68aa289" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34169 } health_report { overall_health: HEALTHY } } }
I20260924 04:13:49.414855 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:49.411961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41062 (local address 127.0.98.3:34169)
0924 04:13:49.412094 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:49.412099 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:49.412115 (+ 16us) server_negotiation.cc:407] Connection header received
0924 04:13:49.412232 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:49.412235 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:49.412283 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:49.412352 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:13:49.412555 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:49.413042 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:49.413967 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:49.414190 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:49.414280 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:49.414498 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:13:49.414563 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:13:49.414612 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:49.414673 (+ 61us) server_negotiation.cc:299] Negotiation successful
0924 04:13:49.414751 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 04:13:49.422688 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:49.420336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48837 (local address 127.0.98.3:34169)
0924 04:13:49.420431 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:49.420434 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:49.420445 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:13:49.420518 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:49.420520 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:49.420555 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:49.420615 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:13:49.420764 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:49.421240 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:49.421946 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:49.422136 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:49.422206 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:49.422357 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:13:49.422413 (+ 56us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:13:49.422456 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:49.422511 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 04:13:49.422566 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":13,"server-negotiator.queue_time_us":35}
I20260924 04:13:49.422690 2610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:49.420259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34169 (local address 127.0.98.3:48837)
0924 04:13:49.420383 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:49.420396 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:49.420458 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:49.420624 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:49.420626 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:49.420631 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:13:49.420704 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:49.420708 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:49.421309 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:49.421311 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:49.421853 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:49.421858 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:49.422210 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:49.422212 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:49.422264 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:49.422285 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:13:49.422463 (+ 178us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:13:49.422466 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 04:13:49.422505 (+ 39us) client_negotiation.cc:240] Negotiation successful
0924 04:13:49.422571 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":3}
I20260924 04:13:49.427695 2417 raft_consensus.cc:1261] T 85dde4779eea43d6a486cd8df9c05c33 P ccbfb74b9f27441ab6aaa9d2e9737ba4 [term 1 FOLLOWER]: Refusing update from remote peer e3be62bcb7e74350a2dc9655b68aa289: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:13:49.427713 2278 raft_consensus.cc:1261] T 85dde4779eea43d6a486cd8df9c05c33 P fcdc2242e68749518a954101b9abc0f0 [term 1 FOLLOWER]: Refusing update from remote peer e3be62bcb7e74350a2dc9655b68aa289: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:13:49.428040 2667 consensus_queue.cc:1035] T 85dde4779eea43d6a486cd8df9c05c33 P e3be62bcb7e74350a2dc9655b68aa289 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:13:49.428208 2634 consensus_queue.cc:1035] T 85dde4779eea43d6a486cd8df9c05c33 P e3be62bcb7e74350a2dc9655b68aa289 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcdc2242e68749518a954101b9abc0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 04:13:49.440407 2655 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 04:13:50.990803 2119 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:13:50.990952 2119 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:13:50.993319 2119 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2fac17aff8b64408b4db746cefeeddbc: Sending DeleteTablet for 3 replicas of tablet 85dde4779eea43d6a486cd8df9c05c33
I20260924 04:13:50.993847 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet 85dde4779eea43d6a486cd8df9c05c33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:13:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48816
I20260924 04:13:50.993959 2538 tablet_service.cc:1514] Processing DeleteTablet for tablet 85dde4779eea43d6a486cd8df9c05c33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:13:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41050
I20260924 04:13:50.994156 2398 tablet_service.cc:1514] Processing DeleteTablet for tablet 85dde4779eea43d6a486cd8df9c05c33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:13:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35604
I20260924 04:13:50.994170 2705 tablet_replica.cc:330] T 85dde4779eea43d6a486cd8df9c05c33 P fcdc2242e68749518a954101b9abc0f0: stopping tablet replica
I20260924 04:13:50.994335 2706 tablet_replica.cc:330] T 85dde4779eea43d6a486cd8df9c05c33 P e3be62bcb7e74350a2dc9655b68aa289: stopping tablet replica
I20260924 04:13:50.994410 2705 raft_consensus.cc:2229] T 85dde4779eea43d6a486cd8df9c05c33 P fcdc2242e68749518a954101b9abc0f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:13:50.994462 2706 raft_consensus.cc:2229] T 85dde4779eea43d6a486cd8df9c05c33 P e3be62bcb7e74350a2dc9655b68aa289 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:13:50.994547 2705 raft_consensus.cc:2258] T 85dde4779eea43d6a486cd8df9c05c33 P fcdc2242e68749518a954101b9abc0f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:13:50.994750 2706 raft_consensus.cc:2258] T 85dde4779eea43d6a486cd8df9c05c33 P e3be62bcb7e74350a2dc9655b68aa289 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:13:50.994913 2707 tablet_replica.cc:330] T 85dde4779eea43d6a486cd8df9c05c33 P ccbfb74b9f27441ab6aaa9d2e9737ba4: stopping tablet replica
I20260924 04:13:50.995131 2707 raft_consensus.cc:2229] T 85dde4779eea43d6a486cd8df9c05c33 P ccbfb74b9f27441ab6aaa9d2e9737ba4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:13:50.995337 2707 raft_consensus.cc:2258] T 85dde4779eea43d6a486cd8df9c05c33 P ccbfb74b9f27441ab6aaa9d2e9737ba4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:13:50.995467 2705 ts_tablet_manager.cc:1905] T 85dde4779eea43d6a486cd8df9c05c33 P fcdc2242e68749518a954101b9abc0f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:13:50.996111 2707 ts_tablet_manager.cc:1905] T 85dde4779eea43d6a486cd8df9c05c33 P ccbfb74b9f27441ab6aaa9d2e9737ba4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:13:50.997582 2706 ts_tablet_manager.cc:1905] T 85dde4779eea43d6a486cd8df9c05c33 P e3be62bcb7e74350a2dc9655b68aa289: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:13:50.998292 2705 ts_tablet_manager.cc:1918] T 85dde4779eea43d6a486cd8df9c05c33 P fcdc2242e68749518a954101b9abc0f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 04:13:50.998377 2705 log.cc:1196] T 85dde4779eea43d6a486cd8df9c05c33 P fcdc2242e68749518a954101b9abc0f0: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/ts-0/wal/wals/85dde4779eea43d6a486cd8df9c05c33
I20260924 04:13:50.998639 2705 ts_tablet_manager.cc:1939] T 85dde4779eea43d6a486cd8df9c05c33 P fcdc2242e68749518a954101b9abc0f0: Deleting consensus metadata
I20260924 04:13:50.998719 2707 ts_tablet_manager.cc:1918] T 85dde4779eea43d6a486cd8df9c05c33 P ccbfb74b9f27441ab6aaa9d2e9737ba4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 04:13:50.998843 2707 log.cc:1196] T 85dde4779eea43d6a486cd8df9c05c33 P ccbfb74b9f27441ab6aaa9d2e9737ba4: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/ts-1/wal/wals/85dde4779eea43d6a486cd8df9c05c33
I20260924 04:13:50.999127 2707 ts_tablet_manager.cc:1939] T 85dde4779eea43d6a486cd8df9c05c33 P ccbfb74b9f27441ab6aaa9d2e9737ba4: Deleting consensus metadata
I20260924 04:13:50.999392 2105 catalog_manager.cc:4841] TS fcdc2242e68749518a954101b9abc0f0 (127.0.98.1:38113): tablet 85dde4779eea43d6a486cd8df9c05c33 (table test-table [id=cc07be245c614358ae98eda01c2dd4a2]) successfully deleted
I20260924 04:13:50.999866 2105 catalog_manager.cc:4841] TS ccbfb74b9f27441ab6aaa9d2e9737ba4 (127.0.98.2:44715): tablet 85dde4779eea43d6a486cd8df9c05c33 (table test-table [id=cc07be245c614358ae98eda01c2dd4a2]) successfully deleted
I20260924 04:13:51.000052 2706 ts_tablet_manager.cc:1918] T 85dde4779eea43d6a486cd8df9c05c33 P e3be62bcb7e74350a2dc9655b68aa289: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 04:13:51.000128 2706 log.cc:1196] T 85dde4779eea43d6a486cd8df9c05c33 P e3be62bcb7e74350a2dc9655b68aa289: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/ts-2/wal/wals/85dde4779eea43d6a486cd8df9c05c33
I20260924 04:13:51.001071 2706 ts_tablet_manager.cc:1939] T 85dde4779eea43d6a486cd8df9c05c33 P e3be62bcb7e74350a2dc9655b68aa289: Deleting consensus metadata
I20260924 04:13:51.001717 2105 catalog_manager.cc:4841] TS e3be62bcb7e74350a2dc9655b68aa289 (127.0.98.3:34169): tablet 85dde4779eea43d6a486cd8df9c05c33 (table test-table [id=cc07be245c614358ae98eda01c2dd4a2]) successfully deleted
Sep 24 04:13:50 dist-test-slave-p401 krb5kdc[2076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223230, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 04:13:51 dist-test-slave-p401 krb5kdc[2076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223230, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:13:51.026822 2715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:51.013436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50542 (local address 127.0.98.62:39493)
0924 04:13:51.013757 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:51.013765 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:51.014071 (+ 306us) server_negotiation.cc:407] Connection header received
0924 04:13:51.014208 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:51.014210 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:51.014259 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:51.014348 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:51.015005 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:51.015628 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:51.016428 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:51.016660 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:51.016786 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:51.023651 (+ 6865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:13:51.023655 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:13:51.023658 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:13:51.023687 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:13:51.025200 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:51.025625 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:51.025628 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:51.025630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:51.025718 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:51.025964 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:51.025967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:51.025969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:51.026164 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:13:51.026285 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:51.026520 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 04:13:51.026646 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":112,"threads_started":1}
I20260924 04:13:51.029460 2119 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50542:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:13:51.029917 2119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:13:51.034356 2398 tablet_service.cc:1467] Processing CreateTablet for tablet f5d787c64c0445f6883bbb392624bd6f (DEFAULT_TABLE table=test-table [id=1e676806538f44f88d05040d20d2a0a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:13:51.034328 2258 tablet_service.cc:1467] Processing CreateTablet for tablet f5d787c64c0445f6883bbb392624bd6f (DEFAULT_TABLE table=test-table [id=1e676806538f44f88d05040d20d2a0a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:13:51.034410 2538 tablet_service.cc:1467] Processing CreateTablet for tablet f5d787c64c0445f6883bbb392624bd6f (DEFAULT_TABLE table=test-table [id=1e676806538f44f88d05040d20d2a0a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:13:51.034623 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5d787c64c0445f6883bbb392624bd6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:13:51.034623 2398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5d787c64c0445f6883bbb392624bd6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:13:51.034701 2538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5d787c64c0445f6883bbb392624bd6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:13:51.037102 2716 tablet_bootstrap.cc:492] T f5d787c64c0445f6883bbb392624bd6f P e3be62bcb7e74350a2dc9655b68aa289: Bootstrap starting.
I20260924 04:13:51.037106 2717 tablet_bootstrap.cc:492] T f5d787c64c0445f6883bbb392624bd6f P ccbfb74b9f27441ab6aaa9d2e9737ba4: Bootstrap starting.
I20260924 04:13:51.037885 2716 tablet_bootstrap.cc:654] T f5d787c64c0445f6883bbb392624bd6f P e3be62bcb7e74350a2dc9655b68aa289: Neither blocks nor log segments found. Creating new log.
I20260924 04:13:51.037894 2717 tablet_bootstrap.cc:654] T f5d787c64c0445f6883bbb392624bd6f P ccbfb74b9f27441ab6aaa9d2e9737ba4: Neither blocks nor log segments found. Creating new log.
I20260924 04:13:51.038712 2718 tablet_bootstrap.cc:492] T f5d787c64c0445f6883bbb392624bd6f P fcdc2242e68749518a954101b9abc0f0: Bootstrap starting.
I20260924 04:13:51.038852 2716 tablet_bootstrap.cc:492] T f5d787c64c0445f6883bbb392624bd6f P e3be62bcb7e74350a2dc9655b68aa289: No bootstrap required, opened a new log
I20260924 04:13:51.038934 2716 ts_tablet_manager.cc:1397] T f5d787c64c0445f6883bbb392624bd6f P e3be62bcb7e74350a2dc9655b68aa289: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:13:51.039323 2716 raft_consensus.cc:348] T f5d787c64c0445f6883bbb392624bd6f P e3be62bcb7e74350a2dc9655b68aa289 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcdc2242e68749518a954101b9abc0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38113 } } peers { permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 } } peers { permanent_uuid: "e3be62bcb7e74350a2dc9655b68aa289" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34169 } }
I20260924 04:13:51.039417 2716 raft_consensus.cc:374] T f5d787c64c0445f6883bbb392624bd6f P e3be62bcb7e74350a2dc9655b68aa289 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:13:51.039424 2718 tablet_bootstrap.cc:654] T f5d787c64c0445f6883bbb392624bd6f P fcdc2242e68749518a954101b9abc0f0: Neither blocks nor log segments found. Creating new log.
I20260924 04:13:51.039458 2716 raft_consensus.cc:729] T f5d787c64c0445f6883bbb392624bd6f P e3be62bcb7e74350a2dc9655b68aa289 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3be62bcb7e74350a2dc9655b68aa289, State: Initialized, Role: FOLLOWER
I20260924 04:13:51.039577 2716 consensus_queue.cc:260] T f5d787c64c0445f6883bbb392624bd6f P e3be62bcb7e74350a2dc9655b68aa289 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcdc2242e68749518a954101b9abc0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38113 } } peers { permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 } } peers { permanent_uuid: "e3be62bcb7e74350a2dc9655b68aa289" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34169 } }
I20260924 04:13:51.039867 2716 ts_tablet_manager.cc:1428] T f5d787c64c0445f6883bbb392624bd6f P e3be62bcb7e74350a2dc9655b68aa289: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:13:51.040287 2717 tablet_bootstrap.cc:492] T f5d787c64c0445f6883bbb392624bd6f P ccbfb74b9f27441ab6aaa9d2e9737ba4: No bootstrap required, opened a new log
I20260924 04:13:51.040407 2717 ts_tablet_manager.cc:1397] T f5d787c64c0445f6883bbb392624bd6f P ccbfb74b9f27441ab6aaa9d2e9737ba4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 04:13:51.040828 2717 raft_consensus.cc:348] T f5d787c64c0445f6883bbb392624bd6f P ccbfb74b9f27441ab6aaa9d2e9737ba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcdc2242e68749518a954101b9abc0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38113 } } peers { permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 } } peers { permanent_uuid: "e3be62bcb7e74350a2dc9655b68aa289" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34169 } }
I20260924 04:13:51.040917 2717 raft_consensus.cc:374] T f5d787c64c0445f6883bbb392624bd6f P ccbfb74b9f27441ab6aaa9d2e9737ba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:13:51.040952 2717 raft_consensus.cc:729] T f5d787c64c0445f6883bbb392624bd6f P ccbfb74b9f27441ab6aaa9d2e9737ba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccbfb74b9f27441ab6aaa9d2e9737ba4, State: Initialized, Role: FOLLOWER
I20260924 04:13:51.041072 2717 consensus_queue.cc:260] T f5d787c64c0445f6883bbb392624bd6f P ccbfb74b9f27441ab6aaa9d2e9737ba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcdc2242e68749518a954101b9abc0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38113 } } peers { permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 } } peers { permanent_uuid: "e3be62bcb7e74350a2dc9655b68aa289" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34169 } }
I20260924 04:13:51.041429 2717 ts_tablet_manager.cc:1428] T f5d787c64c0445f6883bbb392624bd6f P ccbfb74b9f27441ab6aaa9d2e9737ba4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:13:51.041877 2718 tablet_bootstrap.cc:492] T f5d787c64c0445f6883bbb392624bd6f P fcdc2242e68749518a954101b9abc0f0: No bootstrap required, opened a new log
I20260924 04:13:51.041951 2718 ts_tablet_manager.cc:1397] T f5d787c64c0445f6883bbb392624bd6f P fcdc2242e68749518a954101b9abc0f0: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 04:13:51.042325 2718 raft_consensus.cc:348] T f5d787c64c0445f6883bbb392624bd6f P fcdc2242e68749518a954101b9abc0f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcdc2242e68749518a954101b9abc0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38113 } } peers { permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 } } peers { permanent_uuid: "e3be62bcb7e74350a2dc9655b68aa289" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34169 } }
I20260924 04:13:51.042452 2718 raft_consensus.cc:374] T f5d787c64c0445f6883bbb392624bd6f P fcdc2242e68749518a954101b9abc0f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:13:51.042498 2718 raft_consensus.cc:729] T f5d787c64c0445f6883bbb392624bd6f P fcdc2242e68749518a954101b9abc0f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcdc2242e68749518a954101b9abc0f0, State: Initialized, Role: FOLLOWER
I20260924 04:13:51.042608 2718 consensus_queue.cc:260] T f5d787c64c0445f6883bbb392624bd6f P fcdc2242e68749518a954101b9abc0f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcdc2242e68749518a954101b9abc0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38113 } } peers { permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 } } peers { permanent_uuid: "e3be62bcb7e74350a2dc9655b68aa289" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34169 } }
I20260924 04:13:51.042924 2718 ts_tablet_manager.cc:1428] T f5d787c64c0445f6883bbb392624bd6f P fcdc2242e68749518a954101b9abc0f0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:13:51.098577 2722 raft_consensus.cc:482] T f5d787c64c0445f6883bbb392624bd6f P ccbfb74b9f27441ab6aaa9d2e9737ba4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:13:51.098767 2722 raft_consensus.cc:504] T f5d787c64c0445f6883bbb392624bd6f P ccbfb74b9f27441ab6aaa9d2e9737ba4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcdc2242e68749518a954101b9abc0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38113 } } peers { permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 } } peers { permanent_uuid: "e3be62bcb7e74350a2dc9655b68aa289" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34169 } }
I20260924 04:13:51.099923 2722 leader_election.cc:288] T f5d787c64c0445f6883bbb392624bd6f P ccbfb74b9f27441ab6aaa9d2e9737ba4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fcdc2242e68749518a954101b9abc0f0 (127.0.98.1:38113), e3be62bcb7e74350a2dc9655b68aa289 (127.0.98.3:34169)
I20260924 04:13:51.104889 2727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:51.100741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42389 (local address 127.0.98.3:34169)
0924 04:13:51.101028 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:51.101039 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:51.101050 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:13:51.101095 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:51.101098 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:51.101143 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:51.101228 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:13:51.101436 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:51.102178 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:51.103557 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:51.104201 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:51.104317 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:51.104381 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:51.104658 (+ 277us) server_negotiation.cc:299] Negotiation successful
0924 04:13:51.104780 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":91,"threads_started":1}
I20260924 04:13:51.104872 2725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:51.100269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58859 (local address 127.0.98.1:38113)
0924 04:13:51.100540 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:51.100586 (+ 46us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:51.100603 (+ 17us) server_negotiation.cc:407] Connection header received
0924 04:13:51.100702 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:51.100705 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:51.100757 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:51.100859 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:13:51.101128 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:51.101946 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:51.103533 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:51.104201 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:51.104317 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:51.104381 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:51.104658 (+ 277us) server_negotiation.cc:299] Negotiation successful
0924 04:13:51.104734 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":95,"threads_started":1}
I20260924 04:13:51.104957 2726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:51.100589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34169 (local address 127.0.98.2:42389)
0924 04:13:51.100877 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:51.100889 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:51.100970 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:51.101246 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:51.101248 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:51.101262 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:13:51.101346 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:51.101349 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:51.102267 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:51.102271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:51.103443 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:51.103450 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:51.104344 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:51.104348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:51.104511 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:51.104569 (+ 58us) client_negotiation.cc:769] Sending connection context
0924 04:13:51.104636 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 04:13:51.104768 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":84,"thread_start_us":80,"threads_started":1}
I20260924 04:13:51.104955 2724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:51.100160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38113 (local address 127.0.98.2:58859)
0924 04:13:51.100487 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:51.100501 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:51.100601 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:51.100878 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:51.100880 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:51.100900 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:13:51.101031 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:51.101037 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:51.102051 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:51.102053 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:51.103384 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:51.103394 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:51.104334 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:51.104339 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:51.104520 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:51.104561 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 04:13:51.104637 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 04:13:51.104770 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":77,"thread_start_us":132,"threads_started":1}
I20260924 04:13:51.105410 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5d787c64c0445f6883bbb392624bd6f" candidate_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcdc2242e68749518a954101b9abc0f0" is_pre_election: true
I20260924 04:13:51.105410 2558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5d787c64c0445f6883bbb392624bd6f" candidate_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3be62bcb7e74350a2dc9655b68aa289" is_pre_election: true
I20260924 04:13:51.105544 2277 raft_consensus.cc:2454] T f5d787c64c0445f6883bbb392624bd6f P fcdc2242e68749518a954101b9abc0f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccbfb74b9f27441ab6aaa9d2e9737ba4 in term 0.
I20260924 04:13:51.105551 2558 raft_consensus.cc:2454] T f5d787c64c0445f6883bbb392624bd6f P e3be62bcb7e74350a2dc9655b68aa289 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccbfb74b9f27441ab6aaa9d2e9737ba4 in term 0.
I20260924 04:13:51.105890 2349 leader_election.cc:302] T f5d787c64c0445f6883bbb392624bd6f P ccbfb74b9f27441ab6aaa9d2e9737ba4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ccbfb74b9f27441ab6aaa9d2e9737ba4, fcdc2242e68749518a954101b9abc0f0; no voters:
I20260924 04:13:51.106115 2722 raft_consensus.cc:2781] T f5d787c64c0445f6883bbb392624bd6f P ccbfb74b9f27441ab6aaa9d2e9737ba4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:13:51.106179 2722 raft_consensus.cc:482] T f5d787c64c0445f6883bbb392624bd6f P ccbfb74b9f27441ab6aaa9d2e9737ba4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:13:51.106206 2722 raft_consensus.cc:3037] T f5d787c64c0445f6883bbb392624bd6f P ccbfb74b9f27441ab6aaa9d2e9737ba4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:13:51.107733 2722 raft_consensus.cc:504] T f5d787c64c0445f6883bbb392624bd6f P ccbfb74b9f27441ab6aaa9d2e9737ba4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcdc2242e68749518a954101b9abc0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38113 } } peers { permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 } } peers { permanent_uuid: "e3be62bcb7e74350a2dc9655b68aa289" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34169 } }
I20260924 04:13:51.108058 2722 leader_election.cc:288] T f5d787c64c0445f6883bbb392624bd6f P ccbfb74b9f27441ab6aaa9d2e9737ba4 [CANDIDATE]: Term 1 election: Requested vote from peers fcdc2242e68749518a954101b9abc0f0 (127.0.98.1:38113), e3be62bcb7e74350a2dc9655b68aa289 (127.0.98.3:34169)
I20260924 04:13:51.108410 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5d787c64c0445f6883bbb392624bd6f" candidate_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcdc2242e68749518a954101b9abc0f0"
I20260924 04:13:51.108474 2558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5d787c64c0445f6883bbb392624bd6f" candidate_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3be62bcb7e74350a2dc9655b68aa289"
I20260924 04:13:51.108525 2277 raft_consensus.cc:3037] T f5d787c64c0445f6883bbb392624bd6f P fcdc2242e68749518a954101b9abc0f0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:13:51.108584 2558 raft_consensus.cc:3037] T f5d787c64c0445f6883bbb392624bd6f P e3be62bcb7e74350a2dc9655b68aa289 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:13:51.109531 2558 raft_consensus.cc:2454] T f5d787c64c0445f6883bbb392624bd6f P e3be62bcb7e74350a2dc9655b68aa289 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccbfb74b9f27441ab6aaa9d2e9737ba4 in term 1.
I20260924 04:13:51.109557 2277 raft_consensus.cc:2454] T f5d787c64c0445f6883bbb392624bd6f P fcdc2242e68749518a954101b9abc0f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccbfb74b9f27441ab6aaa9d2e9737ba4 in term 1.
I20260924 04:13:51.109800 2349 leader_election.cc:302] T f5d787c64c0445f6883bbb392624bd6f P ccbfb74b9f27441ab6aaa9d2e9737ba4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ccbfb74b9f27441ab6aaa9d2e9737ba4, e3be62bcb7e74350a2dc9655b68aa289; no voters:
I20260924 04:13:51.109964 2722 raft_consensus.cc:2781] T f5d787c64c0445f6883bbb392624bd6f P ccbfb74b9f27441ab6aaa9d2e9737ba4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:13:51.110162 2722 raft_consensus.cc:686] T f5d787c64c0445f6883bbb392624bd6f P ccbfb74b9f27441ab6aaa9d2e9737ba4 [term 1 LEADER]: Becoming Leader. State: Replica: ccbfb74b9f27441ab6aaa9d2e9737ba4, State: Running, Role: LEADER
I20260924 04:13:51.110409 2722 consensus_queue.cc:237] T f5d787c64c0445f6883bbb392624bd6f P ccbfb74b9f27441ab6aaa9d2e9737ba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcdc2242e68749518a954101b9abc0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38113 } } peers { permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 } } peers { permanent_uuid: "e3be62bcb7e74350a2dc9655b68aa289" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34169 } }
I20260924 04:13:51.112789 2119 catalog_manager.cc:5495] T f5d787c64c0445f6883bbb392624bd6f P ccbfb74b9f27441ab6aaa9d2e9737ba4 reported cstate change: term changed from 0 to 1, leader changed from <none> to ccbfb74b9f27441ab6aaa9d2e9737ba4 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcdc2242e68749518a954101b9abc0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3be62bcb7e74350a2dc9655b68aa289" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34169 } health_report { overall_health: UNKNOWN } } }
I20260924 04:13:51.143929 2731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:51.140476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35616 (local address 127.0.98.2:44715)
0924 04:13:51.140746 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:51.140752 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:51.140764 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:13:51.140831 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:51.140833 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:51.140881 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:51.140979 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:13:51.141218 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:51.141736 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:51.142572 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:51.142828 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:51.142910 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:51.143110 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:13:51.143606 (+ 496us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:13:51.143688 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:51.143742 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 04:13:51.143801 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":97,"threads_started":1}
I20260924 04:13:51.153597 2727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:51.150678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55005 (local address 127.0.98.3:34169)
0924 04:13:51.150783 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:51.150786 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:51.150857 (+ 71us) server_negotiation.cc:407] Connection header received
0924 04:13:51.150962 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:51.150964 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:51.151007 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:51.151067 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:13:51.151438 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:51.151924 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:51.152758 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:51.152956 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:51.153022 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:51.153226 (+ 204us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:13:51.153289 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:13:51.153332 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:51.153403 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 04:13:51.153446 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 04:13:51.153609 2734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:51.150502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34169 (local address 127.0.98.2:55005)
0924 04:13:51.150784 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:51.150798 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:51.150887 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:51.151127 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:51.151129 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:51.151137 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:13:51.151317 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:51.151324 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:51.152007 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:51.152009 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:51.152675 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:51.152683 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:51.153031 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:51.153034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:51.153095 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:51.153135 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:13:51.153350 (+ 215us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:13:51.153354 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 04:13:51.153395 (+ 41us) client_negotiation.cc:240] Negotiation successful
0924 04:13:51.153455 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":84,"threads_started":1}
I20260924 04:13:51.159657 2558 raft_consensus.cc:1261] T f5d787c64c0445f6883bbb392624bd6f P e3be62bcb7e74350a2dc9655b68aa289 [term 1 FOLLOWER]: Refusing update from remote peer ccbfb74b9f27441ab6aaa9d2e9737ba4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:13:51.159690 2277 raft_consensus.cc:1261] T f5d787c64c0445f6883bbb392624bd6f P fcdc2242e68749518a954101b9abc0f0 [term 1 FOLLOWER]: Refusing update from remote peer ccbfb74b9f27441ab6aaa9d2e9737ba4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:13:51.160111 2728 consensus_queue.cc:1035] T f5d787c64c0445f6883bbb392624bd6f P ccbfb74b9f27441ab6aaa9d2e9737ba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3be62bcb7e74350a2dc9655b68aa289" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:13:51.160295 2722 consensus_queue.cc:1035] T f5d787c64c0445f6883bbb392624bd6f P ccbfb74b9f27441ab6aaa9d2e9737ba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcdc2242e68749518a954101b9abc0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:13:51.177531 2738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:51.173825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44715 (local address 127.0.98.3:46741)
0924 04:13:51.174070 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:51.174085 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:51.174249 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:51.174561 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:51.174564 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:51.174573 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:13:51.174666 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:51.174670 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:51.175439 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:51.175442 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:51.176355 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:51.176368 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:51.176878 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:51.176882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:51.176970 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:51.176996 (+ 26us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:13:51.177244 (+ 248us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:13:51.177250 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 04:13:51.177307 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 04:13:51.177401 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":98,"threads_started":1}
I20260924 04:13:51.177521 2731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:51.174098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46741 (local address 127.0.98.2:44715)
0924 04:13:51.174233 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:51.174237 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:51.174251 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:13:51.174330 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:51.174333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:51.174380 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:51.174445 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:13:51.174762 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:51.175317 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:51.176483 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:51.176763 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:51.176854 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:51.177098 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:13:51.177175 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:13:51.177229 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:51.177314 (+ 85us) server_negotiation.cc:299] Negotiation successful
0924 04:13:51.177376 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:13:51.200009 2120 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:13:51.200163 2120 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:13:51.202301 2120 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2fac17aff8b64408b4db746cefeeddbc: Sending DeleteTablet for 3 replicas of tablet f5d787c64c0445f6883bbb392624bd6f
I20260924 04:13:51.202991 2538 tablet_service.cc:1514] Processing DeleteTablet for tablet f5d787c64c0445f6883bbb392624bd6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:13:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41050
I20260924 04:13:51.203187 2398 tablet_service.cc:1514] Processing DeleteTablet for tablet f5d787c64c0445f6883bbb392624bd6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:13:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35604
I20260924 04:13:51.203179 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet f5d787c64c0445f6883bbb392624bd6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:13:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48816
I20260924 04:13:51.203425 2705 tablet_replica.cc:330] T f5d787c64c0445f6883bbb392624bd6f P fcdc2242e68749518a954101b9abc0f0: stopping tablet replica
I20260924 04:13:51.203428 2707 tablet_replica.cc:330] T f5d787c64c0445f6883bbb392624bd6f P ccbfb74b9f27441ab6aaa9d2e9737ba4: stopping tablet replica
I20260924 04:13:51.203409 2706 tablet_replica.cc:330] T f5d787c64c0445f6883bbb392624bd6f P e3be62bcb7e74350a2dc9655b68aa289: stopping tablet replica
I20260924 04:13:51.203534 2707 raft_consensus.cc:2229] T f5d787c64c0445f6883bbb392624bd6f P ccbfb74b9f27441ab6aaa9d2e9737ba4 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:13:51.203534 2706 raft_consensus.cc:2229] T f5d787c64c0445f6883bbb392624bd6f P e3be62bcb7e74350a2dc9655b68aa289 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:13:51.203550 2705 raft_consensus.cc:2229] T f5d787c64c0445f6883bbb392624bd6f P fcdc2242e68749518a954101b9abc0f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:13:51.203644 2706 raft_consensus.cc:2258] T f5d787c64c0445f6883bbb392624bd6f P e3be62bcb7e74350a2dc9655b68aa289 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:13:51.203662 2705 raft_consensus.cc:2258] T f5d787c64c0445f6883bbb392624bd6f P fcdc2242e68749518a954101b9abc0f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:13:51.203852 2707 raft_consensus.cc:2258] T f5d787c64c0445f6883bbb392624bd6f P ccbfb74b9f27441ab6aaa9d2e9737ba4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:13:51.204494 2120 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50542:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 04:13:51.204588 2705 ts_tablet_manager.cc:1905] T f5d787c64c0445f6883bbb392624bd6f P fcdc2242e68749518a954101b9abc0f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:13:51.204808 2706 ts_tablet_manager.cc:1905] T f5d787c64c0445f6883bbb392624bd6f P e3be62bcb7e74350a2dc9655b68aa289: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 04:13:51.204926 2120 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:13:51.207340 2705 ts_tablet_manager.cc:1918] T f5d787c64c0445f6883bbb392624bd6f P fcdc2242e68749518a954101b9abc0f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:13:51.207398 2706 ts_tablet_manager.cc:1918] T f5d787c64c0445f6883bbb392624bd6f P e3be62bcb7e74350a2dc9655b68aa289: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:13:51.207506 2706 log.cc:1196] T f5d787c64c0445f6883bbb392624bd6f P e3be62bcb7e74350a2dc9655b68aa289: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/ts-2/wal/wals/f5d787c64c0445f6883bbb392624bd6f
I20260924 04:13:51.207543 2705 log.cc:1196] T f5d787c64c0445f6883bbb392624bd6f P fcdc2242e68749518a954101b9abc0f0: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/ts-0/wal/wals/f5d787c64c0445f6883bbb392624bd6f
I20260924 04:13:51.207855 2706 ts_tablet_manager.cc:1939] T f5d787c64c0445f6883bbb392624bd6f P e3be62bcb7e74350a2dc9655b68aa289: Deleting consensus metadata
I20260924 04:13:51.207856 2705 ts_tablet_manager.cc:1939] T f5d787c64c0445f6883bbb392624bd6f P fcdc2242e68749518a954101b9abc0f0: Deleting consensus metadata
I20260924 04:13:51.208611 2105 catalog_manager.cc:4841] TS e3be62bcb7e74350a2dc9655b68aa289 (127.0.98.3:34169): tablet f5d787c64c0445f6883bbb392624bd6f (table test-table [id=1e676806538f44f88d05040d20d2a0a8]) successfully deleted
I20260924 04:13:51.208760 2105 catalog_manager.cc:4841] TS fcdc2242e68749518a954101b9abc0f0 (127.0.98.1:38113): tablet f5d787c64c0445f6883bbb392624bd6f (table test-table [id=1e676806538f44f88d05040d20d2a0a8]) successfully deleted
I20260924 04:13:51.209805 2707 ts_tablet_manager.cc:1905] T f5d787c64c0445f6883bbb392624bd6f P ccbfb74b9f27441ab6aaa9d2e9737ba4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:13:51.210465 2398 tablet_service.cc:1467] Processing CreateTablet for tablet 2f50700bbcfd4daca8f602eb03954927 (DEFAULT_TABLE table=test-table [id=d53d26fede67466c88ff010869c143d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:13:51.210554 2538 tablet_service.cc:1467] Processing CreateTablet for tablet 2f50700bbcfd4daca8f602eb03954927 (DEFAULT_TABLE table=test-table [id=d53d26fede67466c88ff010869c143d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:13:51.210705 2398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f50700bbcfd4daca8f602eb03954927. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:13:51.210788 2538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f50700bbcfd4daca8f602eb03954927. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:13:51.213045 2717 tablet_bootstrap.cc:492] T 2f50700bbcfd4daca8f602eb03954927 P ccbfb74b9f27441ab6aaa9d2e9737ba4: Bootstrap starting.
I20260924 04:13:51.213074 2716 tablet_bootstrap.cc:492] T 2f50700bbcfd4daca8f602eb03954927 P e3be62bcb7e74350a2dc9655b68aa289: Bootstrap starting.
I20260924 04:13:51.213124 2707 ts_tablet_manager.cc:1918] T f5d787c64c0445f6883bbb392624bd6f P ccbfb74b9f27441ab6aaa9d2e9737ba4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:13:51.213176 2707 log.cc:1196] T f5d787c64c0445f6883bbb392624bd6f P ccbfb74b9f27441ab6aaa9d2e9737ba4: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/ts-1/wal/wals/f5d787c64c0445f6883bbb392624bd6f
I20260924 04:13:51.213402 2707 ts_tablet_manager.cc:1939] T f5d787c64c0445f6883bbb392624bd6f P ccbfb74b9f27441ab6aaa9d2e9737ba4: Deleting consensus metadata
I20260924 04:13:51.213850 2716 tablet_bootstrap.cc:654] T 2f50700bbcfd4daca8f602eb03954927 P e3be62bcb7e74350a2dc9655b68aa289: Neither blocks nor log segments found. Creating new log.
I20260924 04:13:51.213948 2717 tablet_bootstrap.cc:654] T 2f50700bbcfd4daca8f602eb03954927 P ccbfb74b9f27441ab6aaa9d2e9737ba4: Neither blocks nor log segments found. Creating new log.
I20260924 04:13:51.214720 2105 catalog_manager.cc:4841] TS ccbfb74b9f27441ab6aaa9d2e9737ba4 (127.0.98.2:44715): tablet f5d787c64c0445f6883bbb392624bd6f (table test-table [id=1e676806538f44f88d05040d20d2a0a8]) successfully deleted
I20260924 04:13:51.215250 2716 tablet_bootstrap.cc:492] T 2f50700bbcfd4daca8f602eb03954927 P e3be62bcb7e74350a2dc9655b68aa289: No bootstrap required, opened a new log
I20260924 04:13:51.215333 2716 ts_tablet_manager.cc:1397] T 2f50700bbcfd4daca8f602eb03954927 P e3be62bcb7e74350a2dc9655b68aa289: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 04:13:51.215600 2717 tablet_bootstrap.cc:492] T 2f50700bbcfd4daca8f602eb03954927 P ccbfb74b9f27441ab6aaa9d2e9737ba4: No bootstrap required, opened a new log
I20260924 04:13:51.215656 2717 ts_tablet_manager.cc:1397] T 2f50700bbcfd4daca8f602eb03954927 P ccbfb74b9f27441ab6aaa9d2e9737ba4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 04:13:51.215694 2716 raft_consensus.cc:348] T 2f50700bbcfd4daca8f602eb03954927 P e3be62bcb7e74350a2dc9655b68aa289 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcdc2242e68749518a954101b9abc0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38113 } } peers { permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 } } peers { permanent_uuid: "e3be62bcb7e74350a2dc9655b68aa289" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34169 } }
I20260924 04:13:51.215807 2716 raft_consensus.cc:374] T 2f50700bbcfd4daca8f602eb03954927 P e3be62bcb7e74350a2dc9655b68aa289 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:13:51.215835 2716 raft_consensus.cc:729] T 2f50700bbcfd4daca8f602eb03954927 P e3be62bcb7e74350a2dc9655b68aa289 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3be62bcb7e74350a2dc9655b68aa289, State: Initialized, Role: FOLLOWER
I20260924 04:13:51.215950 2716 consensus_queue.cc:260] T 2f50700bbcfd4daca8f602eb03954927 P e3be62bcb7e74350a2dc9655b68aa289 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcdc2242e68749518a954101b9abc0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38113 } } peers { permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 } } peers { permanent_uuid: "e3be62bcb7e74350a2dc9655b68aa289" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34169 } }
I20260924 04:13:51.215994 2717 raft_consensus.cc:348] T 2f50700bbcfd4daca8f602eb03954927 P ccbfb74b9f27441ab6aaa9d2e9737ba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcdc2242e68749518a954101b9abc0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38113 } } peers { permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 } } peers { permanent_uuid: "e3be62bcb7e74350a2dc9655b68aa289" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34169 } }
I20260924 04:13:51.216077 2717 raft_consensus.cc:374] T 2f50700bbcfd4daca8f602eb03954927 P ccbfb74b9f27441ab6aaa9d2e9737ba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:13:51.216104 2717 raft_consensus.cc:729] T 2f50700bbcfd4daca8f602eb03954927 P ccbfb74b9f27441ab6aaa9d2e9737ba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccbfb74b9f27441ab6aaa9d2e9737ba4, State: Initialized, Role: FOLLOWER
I20260924 04:13:51.216207 2717 consensus_queue.cc:260] T 2f50700bbcfd4daca8f602eb03954927 P ccbfb74b9f27441ab6aaa9d2e9737ba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcdc2242e68749518a954101b9abc0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38113 } } peers { permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 } } peers { permanent_uuid: "e3be62bcb7e74350a2dc9655b68aa289" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34169 } }
I20260924 04:13:51.216307 2716 ts_tablet_manager.cc:1428] T 2f50700bbcfd4daca8f602eb03954927 P e3be62bcb7e74350a2dc9655b68aa289: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 04:13:51.216454 2717 ts_tablet_manager.cc:1428] T 2f50700bbcfd4daca8f602eb03954927 P ccbfb74b9f27441ab6aaa9d2e9737ba4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:13:51.217336 2258 tablet_service.cc:1467] Processing CreateTablet for tablet 2f50700bbcfd4daca8f602eb03954927 (DEFAULT_TABLE table=test-table [id=d53d26fede67466c88ff010869c143d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:13:51.217605 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f50700bbcfd4daca8f602eb03954927. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:13:51.219964 2718 tablet_bootstrap.cc:492] T 2f50700bbcfd4daca8f602eb03954927 P fcdc2242e68749518a954101b9abc0f0: Bootstrap starting.
I20260924 04:13:51.220806 2718 tablet_bootstrap.cc:654] T 2f50700bbcfd4daca8f602eb03954927 P fcdc2242e68749518a954101b9abc0f0: Neither blocks nor log segments found. Creating new log.
I20260924 04:13:51.221832 2718 tablet_bootstrap.cc:492] T 2f50700bbcfd4daca8f602eb03954927 P fcdc2242e68749518a954101b9abc0f0: No bootstrap required, opened a new log
I20260924 04:13:51.221920 2718 ts_tablet_manager.cc:1397] T 2f50700bbcfd4daca8f602eb03954927 P fcdc2242e68749518a954101b9abc0f0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 04:13:51.222328 2718 raft_consensus.cc:348] T 2f50700bbcfd4daca8f602eb03954927 P fcdc2242e68749518a954101b9abc0f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcdc2242e68749518a954101b9abc0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38113 } } peers { permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 } } peers { permanent_uuid: "e3be62bcb7e74350a2dc9655b68aa289" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34169 } }
I20260924 04:13:51.222442 2718 raft_consensus.cc:374] T 2f50700bbcfd4daca8f602eb03954927 P fcdc2242e68749518a954101b9abc0f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:13:51.222471 2718 raft_consensus.cc:729] T 2f50700bbcfd4daca8f602eb03954927 P fcdc2242e68749518a954101b9abc0f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcdc2242e68749518a954101b9abc0f0, State: Initialized, Role: FOLLOWER
I20260924 04:13:51.222590 2718 consensus_queue.cc:260] T 2f50700bbcfd4daca8f602eb03954927 P fcdc2242e68749518a954101b9abc0f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcdc2242e68749518a954101b9abc0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38113 } } peers { permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 } } peers { permanent_uuid: "e3be62bcb7e74350a2dc9655b68aa289" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34169 } }
I20260924 04:13:51.222846 2718 ts_tablet_manager.cc:1428] T 2f50700bbcfd4daca8f602eb03954927 P fcdc2242e68749518a954101b9abc0f0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 04:13:51.315073 2723 raft_consensus.cc:482] T 2f50700bbcfd4daca8f602eb03954927 P fcdc2242e68749518a954101b9abc0f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:13:51.315207 2723 raft_consensus.cc:504] T 2f50700bbcfd4daca8f602eb03954927 P fcdc2242e68749518a954101b9abc0f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcdc2242e68749518a954101b9abc0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38113 } } peers { permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 } } peers { permanent_uuid: "e3be62bcb7e74350a2dc9655b68aa289" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34169 } }
I20260924 04:13:51.315615 2723 leader_election.cc:288] T 2f50700bbcfd4daca8f602eb03954927 P fcdc2242e68749518a954101b9abc0f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ccbfb74b9f27441ab6aaa9d2e9737ba4 (127.0.98.2:44715), e3be62bcb7e74350a2dc9655b68aa289 (127.0.98.3:34169)
I20260924 04:13:51.316104 2558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f50700bbcfd4daca8f602eb03954927" candidate_uuid: "fcdc2242e68749518a954101b9abc0f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3be62bcb7e74350a2dc9655b68aa289" is_pre_election: true
I20260924 04:13:51.316078 2417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f50700bbcfd4daca8f602eb03954927" candidate_uuid: "fcdc2242e68749518a954101b9abc0f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" is_pre_election: true
I20260924 04:13:51.316244 2558 raft_consensus.cc:2454] T 2f50700bbcfd4daca8f602eb03954927 P e3be62bcb7e74350a2dc9655b68aa289 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fcdc2242e68749518a954101b9abc0f0 in term 0.
I20260924 04:13:51.316246 2417 raft_consensus.cc:2454] T 2f50700bbcfd4daca8f602eb03954927 P ccbfb74b9f27441ab6aaa9d2e9737ba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fcdc2242e68749518a954101b9abc0f0 in term 0.
I20260924 04:13:51.316543 2209 leader_election.cc:302] T 2f50700bbcfd4daca8f602eb03954927 P fcdc2242e68749518a954101b9abc0f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ccbfb74b9f27441ab6aaa9d2e9737ba4, fcdc2242e68749518a954101b9abc0f0; no voters:
I20260924 04:13:51.316737 2723 raft_consensus.cc:2781] T 2f50700bbcfd4daca8f602eb03954927 P fcdc2242e68749518a954101b9abc0f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:13:51.316797 2723 raft_consensus.cc:482] T 2f50700bbcfd4daca8f602eb03954927 P fcdc2242e68749518a954101b9abc0f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:13:51.316838 2723 raft_consensus.cc:3037] T 2f50700bbcfd4daca8f602eb03954927 P fcdc2242e68749518a954101b9abc0f0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:13:51.317797 2723 raft_consensus.cc:504] T 2f50700bbcfd4daca8f602eb03954927 P fcdc2242e68749518a954101b9abc0f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcdc2242e68749518a954101b9abc0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38113 } } peers { permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 } } peers { permanent_uuid: "e3be62bcb7e74350a2dc9655b68aa289" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34169 } }
I20260924 04:13:51.318069 2723 leader_election.cc:288] T 2f50700bbcfd4daca8f602eb03954927 P fcdc2242e68749518a954101b9abc0f0 [CANDIDATE]: Term 1 election: Requested vote from peers ccbfb74b9f27441ab6aaa9d2e9737ba4 (127.0.98.2:44715), e3be62bcb7e74350a2dc9655b68aa289 (127.0.98.3:34169)
I20260924 04:13:51.318425 2417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f50700bbcfd4daca8f602eb03954927" candidate_uuid: "fcdc2242e68749518a954101b9abc0f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4"
I20260924 04:13:51.318516 2558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f50700bbcfd4daca8f602eb03954927" candidate_uuid: "fcdc2242e68749518a954101b9abc0f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3be62bcb7e74350a2dc9655b68aa289"
I20260924 04:13:51.318550 2417 raft_consensus.cc:3037] T 2f50700bbcfd4daca8f602eb03954927 P ccbfb74b9f27441ab6aaa9d2e9737ba4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:13:51.318620 2558 raft_consensus.cc:3037] T 2f50700bbcfd4daca8f602eb03954927 P e3be62bcb7e74350a2dc9655b68aa289 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:13:51.319710 2558 raft_consensus.cc:2454] T 2f50700bbcfd4daca8f602eb03954927 P e3be62bcb7e74350a2dc9655b68aa289 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcdc2242e68749518a954101b9abc0f0 in term 1.
I20260924 04:13:51.319710 2417 raft_consensus.cc:2454] T 2f50700bbcfd4daca8f602eb03954927 P ccbfb74b9f27441ab6aaa9d2e9737ba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcdc2242e68749518a954101b9abc0f0 in term 1.
I20260924 04:13:51.320009 2209 leader_election.cc:302] T 2f50700bbcfd4daca8f602eb03954927 P fcdc2242e68749518a954101b9abc0f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e3be62bcb7e74350a2dc9655b68aa289, fcdc2242e68749518a954101b9abc0f0; no voters:
I20260924 04:13:51.320293 2723 raft_consensus.cc:2781] T 2f50700bbcfd4daca8f602eb03954927 P fcdc2242e68749518a954101b9abc0f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:13:51.320488 2723 raft_consensus.cc:686] T 2f50700bbcfd4daca8f602eb03954927 P fcdc2242e68749518a954101b9abc0f0 [term 1 LEADER]: Becoming Leader. State: Replica: fcdc2242e68749518a954101b9abc0f0, State: Running, Role: LEADER
I20260924 04:13:51.320639 2723 consensus_queue.cc:237] T 2f50700bbcfd4daca8f602eb03954927 P fcdc2242e68749518a954101b9abc0f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcdc2242e68749518a954101b9abc0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38113 } } peers { permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 } } peers { permanent_uuid: "e3be62bcb7e74350a2dc9655b68aa289" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34169 } }
I20260924 04:13:51.322433 2119 catalog_manager.cc:5495] T 2f50700bbcfd4daca8f602eb03954927 P fcdc2242e68749518a954101b9abc0f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to fcdc2242e68749518a954101b9abc0f0 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "fcdc2242e68749518a954101b9abc0f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcdc2242e68749518a954101b9abc0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3be62bcb7e74350a2dc9655b68aa289" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34169 } health_report { overall_health: UNKNOWN } } }
I20260924 04:13:51.350234 2725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:51.347257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48826 (local address 127.0.98.1:38113)
0924 04:13:51.347549 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:51.347552 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:51.347561 (+ 9us) server_negotiation.cc:407] Connection header received
0924 04:13:51.347594 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:51.347596 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:51.347635 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:51.347694 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:13:51.347951 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:51.348504 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:51.349295 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:51.349516 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:51.349599 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:51.349822 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:13:51.349891 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:13:51.349946 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:51.350048 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 04:13:51.350102 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229}
I20260924 04:13:51.355976 2558 raft_consensus.cc:1261] T 2f50700bbcfd4daca8f602eb03954927 P e3be62bcb7e74350a2dc9655b68aa289 [term 1 FOLLOWER]: Refusing update from remote peer fcdc2242e68749518a954101b9abc0f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:13:51.355980 2417 raft_consensus.cc:1261] T 2f50700bbcfd4daca8f602eb03954927 P ccbfb74b9f27441ab6aaa9d2e9737ba4 [term 1 FOLLOWER]: Refusing update from remote peer fcdc2242e68749518a954101b9abc0f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:13:51.356333 2723 consensus_queue.cc:1035] T 2f50700bbcfd4daca8f602eb03954927 P fcdc2242e68749518a954101b9abc0f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3be62bcb7e74350a2dc9655b68aa289" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:13:51.356482 2745 consensus_queue.cc:1035] T 2f50700bbcfd4daca8f602eb03954927 P fcdc2242e68749518a954101b9abc0f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccbfb74b9f27441ab6aaa9d2e9737ba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 04:13:51.368582 2690 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 04:13:52.921489 2119 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:13:52.921628 2119 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:13:52.924083 2119 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2fac17aff8b64408b4db746cefeeddbc: Sending DeleteTablet for 3 replicas of tablet 2f50700bbcfd4daca8f602eb03954927
I20260924 04:13:52.924571 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f50700bbcfd4daca8f602eb03954927 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:13:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48816
I20260924 04:13:52.924640 2398 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f50700bbcfd4daca8f602eb03954927 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:13:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35604
I20260924 04:13:52.924697 2537 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f50700bbcfd4daca8f602eb03954927 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:13:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41050
I20260924 04:13:52.924939 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 2193
I20260924 04:13:52.925693 2774 tablet_replica.cc:330] T 2f50700bbcfd4daca8f602eb03954927 P e3be62bcb7e74350a2dc9655b68aa289: stopping tablet replica
I20260924 04:13:52.925827 2774 raft_consensus.cc:2229] T 2f50700bbcfd4daca8f602eb03954927 P e3be62bcb7e74350a2dc9655b68aa289 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:13:52.925937 2774 raft_consensus.cc:2258] T 2f50700bbcfd4daca8f602eb03954927 P e3be62bcb7e74350a2dc9655b68aa289 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:13:52.926148 2773 tablet_replica.cc:330] T 2f50700bbcfd4daca8f602eb03954927 P ccbfb74b9f27441ab6aaa9d2e9737ba4: stopping tablet replica
I20260924 04:13:52.926259 2773 raft_consensus.cc:2229] T 2f50700bbcfd4daca8f602eb03954927 P ccbfb74b9f27441ab6aaa9d2e9737ba4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:13:52.926363 2773 raft_consensus.cc:2258] T 2f50700bbcfd4daca8f602eb03954927 P ccbfb74b9f27441ab6aaa9d2e9737ba4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:13:52.926868 2774 ts_tablet_manager.cc:1905] T 2f50700bbcfd4daca8f602eb03954927 P e3be62bcb7e74350a2dc9655b68aa289: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:13:52.927640 2773 ts_tablet_manager.cc:1905] T 2f50700bbcfd4daca8f602eb03954927 P ccbfb74b9f27441ab6aaa9d2e9737ba4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:13:52.929073 2774 ts_tablet_manager.cc:1918] T 2f50700bbcfd4daca8f602eb03954927 P e3be62bcb7e74350a2dc9655b68aa289: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 04:13:52.929137 2774 log.cc:1196] T 2f50700bbcfd4daca8f602eb03954927 P e3be62bcb7e74350a2dc9655b68aa289: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/ts-2/wal/wals/2f50700bbcfd4daca8f602eb03954927
I20260924 04:13:52.929420 2774 ts_tablet_manager.cc:1939] T 2f50700bbcfd4daca8f602eb03954927 P e3be62bcb7e74350a2dc9655b68aa289: Deleting consensus metadata
I20260924 04:13:52.930054 2105 catalog_manager.cc:4841] TS e3be62bcb7e74350a2dc9655b68aa289 (127.0.98.3:34169): tablet 2f50700bbcfd4daca8f602eb03954927 (table test-table [id=d53d26fede67466c88ff010869c143d4]) successfully deleted
I20260924 04:13:52.930361 2773 ts_tablet_manager.cc:1918] T 2f50700bbcfd4daca8f602eb03954927 P ccbfb74b9f27441ab6aaa9d2e9737ba4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 04:13:52.930433 2773 log.cc:1196] T 2f50700bbcfd4daca8f602eb03954927 P ccbfb74b9f27441ab6aaa9d2e9737ba4: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790223216265354-392-0/minicluster-data/ts-1/wal/wals/2f50700bbcfd4daca8f602eb03954927
I20260924 04:13:52.930660 2773 ts_tablet_manager.cc:1939] T 2f50700bbcfd4daca8f602eb03954927 P ccbfb74b9f27441ab6aaa9d2e9737ba4: Deleting consensus metadata
I20260924 04:13:52.931298 2105 catalog_manager.cc:4841] TS ccbfb74b9f27441ab6aaa9d2e9737ba4 (127.0.98.2:44715): tablet 2f50700bbcfd4daca8f602eb03954927 (table test-table [id=d53d26fede67466c88ff010869c143d4]) successfully deleted
I20260924 04:13:52.932933 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 2333
W20260924 04:13:52.932932 2105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38113 (error 108)
I20260924 04:13:52.936105 2776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:52.935670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38113 (local address 127.0.0.1:48834)
0924 04:13:52.935926 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:52.936007 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":74,"threads_started":1}
W20260924 04:13:52.936239 2105 catalog_manager.cc:4584] TS fcdc2242e68749518a954101b9abc0f0 (127.0.98.1:38113): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2f50700bbcfd4daca8f602eb03954927: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38113: connect: Connection refused (error 111)
I20260924 04:13:52.939213 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 2473
I20260924 04:13:52.944926 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 2089
2026-09-24T04:13:52Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6263 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:13:52 dist-test-slave-p401 krb5kdc[2779](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:13:52 dist-test-slave-p401 krb5kdc[2779](info): set up 2 sockets
Sep 24 04:13:52 dist-test-slave-p401 krb5kdc[2779](info): commencing operation
krb5kdc: starting...
W20260924 04:13:53.910888 392 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/krb5kdc/test-user.keytab.
Sep 24 04:13:53 dist-test-slave-p401 krb5kdc[2779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:13:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:13:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:13:54.027441 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46317
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46317
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:13:54.143848 2792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:13:54.144099 2792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:13:54.144140 2792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:13:54.148114 2792 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:13:54.148245 2792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:13:54.148283 2792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:13:54.148299 2792 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:13:54.148314 2792 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:13:54.153642 2792 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46317
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:13:54.154649 2792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:13:54.155620 2792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:13:54.164086 2800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:13:54.164089 2797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:13:54.164157 2792 server_base.cc:1034] running on GCE node
W20260924 04:13:54.164081 2798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:13:54.165005 2792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:13:54.166095 2792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:13:54.167276 2792 hybrid_clock.cc:648] HybridClock initialized: now 1790223234167252 us; error 39 us; skew 500 ppm
Sep 24 04:13:54 dist-test-slave-p401 krb5kdc[2779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223234, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:13:54.169924 2792 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:13:54.170799 2792 webserver.cc:460] Webserver started at http://127.0.98.62:35561/ using document root <none> and password file <none>
I20260924 04:13:54.171368 2792 fs_manager.cc:359] Metadata directory not provided
I20260924 04:13:54.171451 2792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:13:54.171648 2792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:13:54.173636 2792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/master-0/data/instance:
uuid: "61649b536f224ef9aa1835f636ebfbad"
format_stamp: "Formatted at 2026-09-24 04:13:54 on dist-test-slave-p401"
server_key: "718cb19e15d0dbe39f690ebd3fb1e2c3"
server_key_iv: "cf2079efdd5dd219df52b4995f4afe02"
server_key_version: "encryptionkey@0"
I20260924 04:13:54.174163 2792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/master-0/wal/instance:
uuid: "61649b536f224ef9aa1835f636ebfbad"
format_stamp: "Formatted at 2026-09-24 04:13:54 on dist-test-slave-p401"
server_key: "718cb19e15d0dbe39f690ebd3fb1e2c3"
server_key_iv: "cf2079efdd5dd219df52b4995f4afe02"
server_key_version: "encryptionkey@0"
I20260924 04:13:54.177785 2792 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 04:13:54.180125 2807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:13:54.181344 2792 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:13:54.181454 2792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/master-0/wal
uuid: "61649b536f224ef9aa1835f636ebfbad"
format_stamp: "Formatted at 2026-09-24 04:13:54 on dist-test-slave-p401"
server_key: "718cb19e15d0dbe39f690ebd3fb1e2c3"
server_key_iv: "cf2079efdd5dd219df52b4995f4afe02"
server_key_version: "encryptionkey@0"
I20260924 04:13:54.181543 2792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:13:54.196951 2792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:13:54.199994 2792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:13:54.200165 2792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:13:54.206713 2792 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46317
I20260924 04:13:54.206790 2859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46317 every 8 connection(s)
I20260924 04:13:54.207696 2792 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:13:54.210456 2860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:13:54.213379 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 2792
I20260924 04:13:54.213496 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/master-0/wal/instance
I20260924 04:13:54.213743 392 external_mini_cluster.cc:1428] Setting key 5ba69bb43ffaf1c9b5432497159bc8e9
I20260924 04:13:54.216357 2860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61649b536f224ef9aa1835f636ebfbad: Bootstrap starting.
I20260924 04:13:54.218803 2860 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61649b536f224ef9aa1835f636ebfbad: Neither blocks nor log segments found. Creating new log.
I20260924 04:13:54.219761 2860 log.cc:824] T 00000000000000000000000000000000 P 61649b536f224ef9aa1835f636ebfbad: Log is configured to *not* fsync() on all Append() calls
I20260924 04:13:54.222071 2860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61649b536f224ef9aa1835f636ebfbad: No bootstrap required, opened a new log
Sep 24 04:13:54 dist-test-slave-p401 krb5kdc[2779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:13:54.225924 2860 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61649b536f224ef9aa1835f636ebfbad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61649b536f224ef9aa1835f636ebfbad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46317 } }
I20260924 04:13:54.226203 2860 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61649b536f224ef9aa1835f636ebfbad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:13:54.226271 2860 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61649b536f224ef9aa1835f636ebfbad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61649b536f224ef9aa1835f636ebfbad, State: Initialized, Role: FOLLOWER
I20260924 04:13:54.226696 2860 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61649b536f224ef9aa1835f636ebfbad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61649b536f224ef9aa1835f636ebfbad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46317 } }
I20260924 04:13:54.226818 2860 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61649b536f224ef9aa1835f636ebfbad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:13:54.226883 2860 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61649b536f224ef9aa1835f636ebfbad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:13:54.226979 2860 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61649b536f224ef9aa1835f636ebfbad [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:13:54.227993 2860 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61649b536f224ef9aa1835f636ebfbad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61649b536f224ef9aa1835f636ebfbad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46317 } }
I20260924 04:13:54.228320 2860 leader_election.cc:302] T 00000000000000000000000000000000 P 61649b536f224ef9aa1835f636ebfbad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61649b536f224ef9aa1835f636ebfbad; no voters:
I20260924 04:13:54.228351 2863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:54.215030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43108 (local address 127.0.98.62:46317)
0924 04:13:54.215424 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:54.215439 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:54.215472 (+ 33us) server_negotiation.cc:407] Connection header received
0924 04:13:54.216077 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:54.216093 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:54.216420 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:54.216707 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:54.217542 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:54.218539 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:54.219709 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:54.220296 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:54.220467 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:54.223000 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:13:54.223006 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:13:54.223022 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:13:54.223052 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:13:54.225555 (+ 2503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:54.225957 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:54.225963 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:54.225968 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:54.226100 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:54.226375 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:54.226378 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:54.226379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:54.226792 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:13:54.226976 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:54.227359 (+ 383us) server_negotiation.cc:299] Negotiation successful
0924 04:13:54.227628 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":137,"threads_started":1}
I20260924 04:13:54.228628 2860 leader_election.cc:288] T 00000000000000000000000000000000 P 61649b536f224ef9aa1835f636ebfbad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:13:54.228730 2865 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61649b536f224ef9aa1835f636ebfbad [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:13:54.228878 2865 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61649b536f224ef9aa1835f636ebfbad [term 1 LEADER]: Becoming Leader. State: Replica: 61649b536f224ef9aa1835f636ebfbad, State: Running, Role: LEADER
I20260924 04:13:54.229235 2865 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61649b536f224ef9aa1835f636ebfbad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61649b536f224ef9aa1835f636ebfbad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46317 } }
I20260924 04:13:54.230890 2866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61649b536f224ef9aa1835f636ebfbad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61649b536f224ef9aa1835f636ebfbad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61649b536f224ef9aa1835f636ebfbad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46317 } } }
I20260924 04:13:54.231014 2866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61649b536f224ef9aa1835f636ebfbad [sys.catalog]: This master's current role is: LEADER
I20260924 04:13:54.231285 2867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61649b536f224ef9aa1835f636ebfbad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61649b536f224ef9aa1835f636ebfbad. Latest consensus state: current_term: 1 leader_uuid: "61649b536f224ef9aa1835f636ebfbad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61649b536f224ef9aa1835f636ebfbad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46317 } } }
I20260924 04:13:54.231400 2867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61649b536f224ef9aa1835f636ebfbad [sys.catalog]: This master's current role is: LEADER
I20260924 04:13:54.231500 2860 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61649b536f224ef9aa1835f636ebfbad [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:13:54.231951 2869 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:13:54.234963 2869 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:13:54.241001 2869 catalog_manager.cc:1347] Generated new cluster ID: a590f2438a5b4a519aa90590250120ad
I20260924 04:13:54.241101 2869 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:13:54.256706 2869 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:13:54.258055 2869 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:13:54.272369 2869 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61649b536f224ef9aa1835f636ebfbad: Generated new TSK 0
I20260924 04:13:54.273103 2869 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:13:54.311055 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46317
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:13:54.435999 2886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:13:54.436264 2886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:13:54.436304 2886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:13:54.440315 2886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:13:54.440382 2886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:13:54.440460 2886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:13:54.445691 2886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:13:54.446776 2886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:13:54.447800 2886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:13:54.456974 2892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:13:54.457072 2891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:13:54.457159 2886 server_base.cc:1034] running on GCE node
W20260924 04:13:54.457093 2894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:13:54.457686 2886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:13:54.458297 2886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:13:54.459446 2886 hybrid_clock.cc:648] HybridClock initialized: now 1790223234459426 us; error 30 us; skew 500 ppm
Sep 24 04:13:54 dist-test-slave-p401 krb5kdc[2779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223234, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:13:54.462054 2886 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:13:54.462877 2886 webserver.cc:460] Webserver started at http://127.0.98.1:43379/ using document root <none> and password file <none>
I20260924 04:13:54.463433 2886 fs_manager.cc:359] Metadata directory not provided
I20260924 04:13:54.463498 2886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:13:54.463689 2886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:13:54.465648 2886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-0/data/instance:
uuid: "1646c9c4a0df4f65916409ff15d7c16a"
format_stamp: "Formatted at 2026-09-24 04:13:54 on dist-test-slave-p401"
server_key: "64b6d51917a2efd5807972525a162450"
server_key_iv: "829e4d6a580e7a14a3bccc2b9a6adb04"
server_key_version: "encryptionkey@0"
I20260924 04:13:54.466180 2886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1646c9c4a0df4f65916409ff15d7c16a"
format_stamp: "Formatted at 2026-09-24 04:13:54 on dist-test-slave-p401"
server_key: "64b6d51917a2efd5807972525a162450"
server_key_iv: "829e4d6a580e7a14a3bccc2b9a6adb04"
server_key_version: "encryptionkey@0"
I20260924 04:13:54.469664 2886 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 04:13:54.471976 2901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:13:54.473099 2886 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:13:54.473204 2886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-0/wal
uuid: "1646c9c4a0df4f65916409ff15d7c16a"
format_stamp: "Formatted at 2026-09-24 04:13:54 on dist-test-slave-p401"
server_key: "64b6d51917a2efd5807972525a162450"
server_key_iv: "829e4d6a580e7a14a3bccc2b9a6adb04"
server_key_version: "encryptionkey@0"
I20260924 04:13:54.473289 2886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:13:54.489853 2886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:13:54.492748 2886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:13:54.492926 2886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:13:54.493466 2886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:13:54.494234 2886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:13:54.494279 2886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:13:54.494329 2886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:13:54.494357 2886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:13:54.502506 2886 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36227
I20260924 04:13:54.502540 3014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36227 every 8 connection(s)
I20260924 04:13:54.503393 2886 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:13:54.507531 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 2886
I20260924 04:13:54.507650 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:13:54.507956 392 external_mini_cluster.cc:1428] Setting key 4e9cff333d88c5ffaa535878703c0e7a
Sep 24 04:13:54 dist-test-slave-p401 krb5kdc[2779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223234, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:13:54.517320 2863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:54.505061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60065 (local address 127.0.98.62:46317)
0924 04:13:54.505224 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:54.505228 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:54.505971 (+ 743us) server_negotiation.cc:407] Connection header received
0924 04:13:54.507373 (+ 1402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:54.507376 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:54.507425 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:54.507502 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:54.508901 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:54.509476 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:54.511084 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:54.511466 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:54.511566 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:54.514010 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:13:54.514013 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:13:54.514015 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:13:54.514038 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:13:54.515333 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:54.515888 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:54.515890 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:54.515892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:54.515965 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:54.516314 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:54.516316 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:54.516317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:54.516478 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:13:54.516549 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:54.517118 (+ 569us) server_negotiation.cc:299] Negotiation successful
0924 04:13:54.517208 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:13:54.518234 3017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:54.505366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46317 (local address 127.0.98.1:60065)
0924 04:13:54.505828 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:54.505870 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:54.507115 (+ 1245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:54.507727 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:54.507736 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:54.508195 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:13:54.508764 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:54.508784 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:54.509594 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:54.509598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:54.510945 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:54.510958 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:54.511615 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:54.511619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:54.511761 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:54.512308 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:13:54.512324 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:13:54.513855 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:13:54.515433 (+ 1578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:54.515439 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:54.515454 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:54.515782 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:54.516046 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:54.516050 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:54.516052 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:54.516222 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:54.516572 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:13:54.516578 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:13:54.516893 (+ 315us) client_negotiation.cc:769] Sending connection context
0924 04:13:54.517103 (+ 210us) client_negotiation.cc:240] Negotiation successful
0924 04:13:54.517423 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":99,"threads_started":1}
I20260924 04:13:54.519368 3015 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46317
I20260924 04:13:54.519568 3015 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:13:54.520057 3015 heartbeater.cc:511] Master 127.0.98.62:46317 requested a full tablet report, sending...
I20260924 04:13:54.521562 2824 ts_manager.cc:194] Registered new tserver with Master: 1646c9c4a0df4f65916409ff15d7c16a (127.0.98.1:36227)
I20260924 04:13:54.522634 2824 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60065
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:13:54.533885 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46317
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:13:54.638439 3020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:13:54.638701 3020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:13:54.638761 3020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:13:54.643153 3020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:13:54.643216 3020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:13:54.643322 3020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:13:54.648478 3020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:13:54.649533 3020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:13:54.650504 3020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:13:54.659493 3028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:13:54.659518 3025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:13:54.659615 3020 server_base.cc:1034] running on GCE node
W20260924 04:13:54.659564 3026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:13:54.660087 3020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:13:54.660667 3020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:13:54.661836 3020 hybrid_clock.cc:648] HybridClock initialized: now 1790223234661823 us; error 26 us; skew 500 ppm
Sep 24 04:13:54 dist-test-slave-p401 krb5kdc[2779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223234, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:13:54.664404 3020 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:13:54.665234 3020 webserver.cc:460] Webserver started at http://127.0.98.2:39741/ using document root <none> and password file <none>
I20260924 04:13:54.665740 3020 fs_manager.cc:359] Metadata directory not provided
I20260924 04:13:54.665805 3020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:13:54.665982 3020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:13:54.667968 3020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-1/data/instance:
uuid: "5d312a35b17d4d20a9844adda46a88d9"
format_stamp: "Formatted at 2026-09-24 04:13:54 on dist-test-slave-p401"
server_key: "a1a0a6b9b8867a9ca4586d839dd4db1f"
server_key_iv: "5986d76ee21055dbc3670a5b54c56c98"
server_key_version: "encryptionkey@0"
I20260924 04:13:54.668512 3020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5d312a35b17d4d20a9844adda46a88d9"
format_stamp: "Formatted at 2026-09-24 04:13:54 on dist-test-slave-p401"
server_key: "a1a0a6b9b8867a9ca4586d839dd4db1f"
server_key_iv: "5986d76ee21055dbc3670a5b54c56c98"
server_key_version: "encryptionkey@0"
I20260924 04:13:54.671922 3020 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:13:54.674134 3035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:13:54.675308 3020 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:13:54.675422 3020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-1/wal
uuid: "5d312a35b17d4d20a9844adda46a88d9"
format_stamp: "Formatted at 2026-09-24 04:13:54 on dist-test-slave-p401"
server_key: "a1a0a6b9b8867a9ca4586d839dd4db1f"
server_key_iv: "5986d76ee21055dbc3670a5b54c56c98"
server_key_version: "encryptionkey@0"
I20260924 04:13:54.675506 3020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:13:54.716394 3020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:13:54.719329 3020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:13:54.719504 3020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:13:54.720118 3020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:13:54.720996 3020 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:13:54.721058 3020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:13:54.721114 3020 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:13:54.721145 3020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:13:54.731127 3020 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39523
I20260924 04:13:54.731189 3148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39523 every 8 connection(s)
I20260924 04:13:54.732183 3020 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:13:54.739629 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 3020
I20260924 04:13:54.739711 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:13:54.739957 392 external_mini_cluster.cc:1428] Setting key 8b8a8c9392ac50b68e7247a9b7fef135
Sep 24 04:13:54 dist-test-slave-p401 krb5kdc[2779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223234, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:13:54.745514 2863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:54.733913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41407 (local address 127.0.98.62:46317)
0924 04:13:54.734055 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:54.734060 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:54.734884 (+ 824us) server_negotiation.cc:407] Connection header received
0924 04:13:54.736210 (+ 1326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:54.736212 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:54.736258 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:54.736328 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:54.737429 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:54.737936 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:54.739021 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:54.739271 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:54.739364 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:54.741992 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:13:54.741995 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:13:54.741997 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:13:54.742019 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:13:54.743462 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:54.744028 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:54.744031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:54.744033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:54.744112 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:54.744428 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:54.744431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:54.744433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:54.744606 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:13:54.744692 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:54.745243 (+ 551us) server_negotiation.cc:299] Negotiation successful
0924 04:13:54.745360 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 04:13:54.746287 3151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:54.734299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46317 (local address 127.0.98.2:41407)
0924 04:13:54.734744 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:54.734785 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:54.736018 (+ 1233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:54.736473 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:54.736480 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:54.736835 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:13:54.737320 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:54.737334 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:54.738041 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:54.738044 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:54.738923 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:54.738931 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:54.739416 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:54.739420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:54.739559 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:54.740166 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:13:54.740183 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:13:54.741803 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:13:54.743587 (+ 1784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:54.743591 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:54.743604 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:54.743908 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:54.744202 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:54.744205 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:54.744208 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:54.744343 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:54.744728 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:13:54.744734 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:13:54.745019 (+ 285us) client_negotiation.cc:769] Sending connection context
0924 04:13:54.745182 (+ 163us) client_negotiation.cc:240] Negotiation successful
0924 04:13:54.745503 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":103,"threads_started":1}
I20260924 04:13:54.747366 3149 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46317
I20260924 04:13:54.747589 3149 heartbeater.cc:464] Registering TS with master...
I20260924 04:13:54.748061 3149 heartbeater.cc:511] Master 127.0.98.62:46317 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:13:54.749107 2824 ts_manager.cc:194] Registered new tserver with Master: 5d312a35b17d4d20a9844adda46a88d9 (127.0.98.2:39523)
I20260924 04:13:54.749820 2824 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41407
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:13:54.766179 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46317
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:13:54.874156 3154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:13:54.874432 3154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:13:54.874475 3154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:13:54.878615 3154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:13:54.878677 3154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:13:54.878742 3154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:13:54.884110 3154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:13:54.885232 3154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:13:54.886271 3154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:13:54.895514 3160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:13:54.895496 3159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:13:54.895601 3154 server_base.cc:1034] running on GCE node
W20260924 04:13:54.895527 3162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:13:54.896157 3154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:13:54.896790 3154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:13:54.897965 3154 hybrid_clock.cc:648] HybridClock initialized: now 1790223234897943 us; error 38 us; skew 500 ppm
Sep 24 04:13:54 dist-test-slave-p401 krb5kdc[2779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223234, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:13:54.900664 3154 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:13:54.901602 3154 webserver.cc:460] Webserver started at http://127.0.98.3:39657/ using document root <none> and password file <none>
I20260924 04:13:54.902137 3154 fs_manager.cc:359] Metadata directory not provided
I20260924 04:13:54.902217 3154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:13:54.902413 3154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:13:54.904498 3154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-2/data/instance:
uuid: "84f38148510942d698cd037a762f0bf7"
format_stamp: "Formatted at 2026-09-24 04:13:54 on dist-test-slave-p401"
server_key: "a31d0a57d93fb876ab41f05c2e49079a"
server_key_iv: "c126f779a2402376798be6f76af5fecb"
server_key_version: "encryptionkey@0"
I20260924 04:13:54.905054 3154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-2/wal/instance:
uuid: "84f38148510942d698cd037a762f0bf7"
format_stamp: "Formatted at 2026-09-24 04:13:54 on dist-test-slave-p401"
server_key: "a31d0a57d93fb876ab41f05c2e49079a"
server_key_iv: "c126f779a2402376798be6f76af5fecb"
server_key_version: "encryptionkey@0"
I20260924 04:13:54.909057 3154 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:13:54.911557 3169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:13:54.912838 3154 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 04:13:54.912957 3154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-2/wal
uuid: "84f38148510942d698cd037a762f0bf7"
format_stamp: "Formatted at 2026-09-24 04:13:54 on dist-test-slave-p401"
server_key: "a31d0a57d93fb876ab41f05c2e49079a"
server_key_iv: "c126f779a2402376798be6f76af5fecb"
server_key_version: "encryptionkey@0"
I20260924 04:13:54.913061 3154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:13:54.925372 3154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:13:54.928103 3154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:13:54.928274 3154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:13:54.928845 3154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:13:54.929626 3154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:13:54.929682 3154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:13:54.929735 3154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:13:54.929760 3154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:13:54.938271 3154 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38455
I20260924 04:13:54.938290 3282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38455 every 8 connection(s)
I20260924 04:13:54.939162 3154 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:13:54.942063 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 3154
I20260924 04:13:54.942175 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:13:54.942435 392 external_mini_cluster.cc:1428] Setting key 8937207df315925c816bda7604632db0
Sep 24 04:13:54 dist-test-slave-p401 krb5kdc[2779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223234, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:13:54.952941 2863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:54.940964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48845 (local address 127.0.98.62:46317)
0924 04:13:54.941128 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:54.941132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:54.942017 (+ 885us) server_negotiation.cc:407] Connection header received
0924 04:13:54.943580 (+ 1563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:54.943582 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:54.943634 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:54.943711 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:54.945094 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:54.945606 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:54.946711 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:54.946932 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:54.947011 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:54.949606 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:13:54.949611 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:13:54.949614 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:13:54.949645 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:13:54.950956 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:54.951562 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:54.951566 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:54.951567 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:54.951654 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:54.951956 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:54.951958 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:54.951960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:54.952135 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:13:54.952214 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:54.952708 (+ 494us) server_negotiation.cc:299] Negotiation successful
0924 04:13:54.952815 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 04:13:54.953756 3285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:54.941343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46317 (local address 127.0.98.3:48845)
0924 04:13:54.941856 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:54.941900 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:54.943382 (+ 1482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:54.943922 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:54.943931 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:54.944413 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:13:54.944946 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:54.944960 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:54.945702 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:54.945704 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:54.946592 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:54.946601 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:54.947074 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:54.947077 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:54.947258 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:54.947875 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:13:54.947893 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:13:54.949403 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:13:54.951079 (+ 1676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:54.951084 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:54.951098 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:54.951447 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:54.951729 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:54.951731 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:54.951733 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:54.951886 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:54.952223 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:13:54.952228 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:13:54.952486 (+ 258us) client_negotiation.cc:769] Sending connection context
0924 04:13:54.952639 (+ 153us) client_negotiation.cc:240] Negotiation successful
0924 04:13:54.952933 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":103,"threads_started":1}
I20260924 04:13:54.954934 3283 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46317
I20260924 04:13:54.955206 3283 heartbeater.cc:464] Registering TS with master...
I20260924 04:13:54.955881 3283 heartbeater.cc:511] Master 127.0.98.62:46317 requested a full tablet report, sending...
I20260924 04:13:54.956877 2824 ts_manager.cc:194] Registered new tserver with Master: 84f38148510942d698cd037a762f0bf7 (127.0.98.3:38455)
I20260924 04:13:54.957409 2824 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48845
I20260924 04:13:54.968371 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 04:13:54.981010 2863 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:13:54.979537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43110 (local address 127.0.98.62:46317)
0924 04:13:54.979730 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:54.979735 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:54.979857 (+ 122us) server_negotiation.cc:407] Connection header received
0924 04:13:54.979991 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:54.979994 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:54.980056 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:54.980128 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:54.980868 (+ 740us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43110: BlockingRecv error: recv got EOF from 127.0.0.1:43110 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 04:13:54.981979 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 2886
I20260924 04:13:54.986739 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 3020
I20260924 04:13:54.991088 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 3154
I20260924 04:13:54.995570 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 2792
2026-09-24T04:13:54Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2049 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790223216265354-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:13:55 dist-test-slave-p401 krb5kdc[3295](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:13:55 dist-test-slave-p401 krb5kdc[3295](info): set up 2 sockets
Sep 24 04:13:55 dist-test-slave-p401 krb5kdc[3295](info): commencing operation
krb5kdc: starting...
W20260924 04:13:55.947049 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790223216265354-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790223216265354-392-0/krb5kdc/test-user.keytab.
Sep 24 04:13:56 dist-test-slave-p401 krb5kdc[3295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223236, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:13:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:13:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:13:56.061494 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790223216265354-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790223216265354-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790223216265354-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40019
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:13:56.166576 3308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:13:56.166841 3308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:13:56.166895 3308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:13:56.170881 3308 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:13:56.170950 3308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:13:56.170976 3308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:13:56.170996 3308 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:13:56.171018 3308 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:13:56.176364 3308 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790223216265354-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790223216265354-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790223216265354-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:13:56.177441 3308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:13:56.178457 3308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:13:56.186784 3314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:13:56.186786 3313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:13:56.187044 3308 server_base.cc:1034] running on GCE node
W20260924 04:13:56.186905 3316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:13:56.187757 3308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:13:56.188859 3308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:13:56.190021 3308 hybrid_clock.cc:648] HybridClock initialized: now 1790223236190004 us; error 35 us; skew 500 ppm
Sep 24 04:13:56 dist-test-slave-p401 krb5kdc[3295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223236, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:13:56.196463 3308 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:13:56.197373 3308 webserver.cc:460] Webserver started at http://127.0.98.62:34307/ using document root <none> and password file <none>
I20260924 04:13:56.197892 3308 fs_manager.cc:359] Metadata directory not provided
I20260924 04:13:56.197947 3308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:13:56.198119 3308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:13:56.200202 3308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790223216265354-392-0/minicluster-data/master-0/data/instance:
uuid: "34bffd1f0ca14092bc3cf0510b04b1bc"
format_stamp: "Formatted at 2026-09-24 04:13:56 on dist-test-slave-p401"
server_key: "3386038612a883431065c131a9553c42"
server_key_iv: "b338666617da0a9b75edea28841b6dc1"
server_key_version: "encryptionkey@0"
I20260924 04:13:56.200734 3308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790223216265354-392-0/minicluster-data/master-0/wal/instance:
uuid: "34bffd1f0ca14092bc3cf0510b04b1bc"
format_stamp: "Formatted at 2026-09-24 04:13:56 on dist-test-slave-p401"
server_key: "3386038612a883431065c131a9553c42"
server_key_iv: "b338666617da0a9b75edea28841b6dc1"
server_key_version: "encryptionkey@0"
I20260924 04:13:56.204329 3308 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:13:56.206658 3323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:13:56.207964 3308 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:13:56.208088 3308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790223216265354-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790223216265354-392-0/minicluster-data/master-0/wal
uuid: "34bffd1f0ca14092bc3cf0510b04b1bc"
format_stamp: "Formatted at 2026-09-24 04:13:56 on dist-test-slave-p401"
server_key: "3386038612a883431065c131a9553c42"
server_key_iv: "b338666617da0a9b75edea28841b6dc1"
server_key_version: "encryptionkey@0"
I20260924 04:13:56.208189 3308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790223216265354-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790223216265354-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790223216265354-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:13:56.218730 3308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:13:56.221611 3308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:13:56.221802 3308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:13:56.228333 3308 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40019
I20260924 04:13:56.228582 3375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40019 every 8 connection(s)
I20260924 04:13:56.229242 3308 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:13:56.231895 3376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:13:56.236682 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 3308
I20260924 04:13:56.236806 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790223216265354-392-0/minicluster-data/master-0/wal/instance
I20260924 04:13:56.237087 392 external_mini_cluster.cc:1428] Setting key 19ac29ac3882a9693a4feb1b837f1668
I20260924 04:13:56.237519 3376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34bffd1f0ca14092bc3cf0510b04b1bc: Bootstrap starting.
I20260924 04:13:56.239825 3376 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34bffd1f0ca14092bc3cf0510b04b1bc: Neither blocks nor log segments found. Creating new log.
I20260924 04:13:56.240512 3376 log.cc:824] T 00000000000000000000000000000000 P 34bffd1f0ca14092bc3cf0510b04b1bc: Log is configured to *not* fsync() on all Append() calls
I20260924 04:13:56.242206 3376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34bffd1f0ca14092bc3cf0510b04b1bc: No bootstrap required, opened a new log
Sep 24 04:13:56 dist-test-slave-p401 krb5kdc[3295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223236, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:13:56.245585 3376 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34bffd1f0ca14092bc3cf0510b04b1bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34bffd1f0ca14092bc3cf0510b04b1bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40019 } }
I20260924 04:13:56.245796 3376 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34bffd1f0ca14092bc3cf0510b04b1bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:13:56.245852 3376 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34bffd1f0ca14092bc3cf0510b04b1bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34bffd1f0ca14092bc3cf0510b04b1bc, State: Initialized, Role: FOLLOWER
I20260924 04:13:56.246296 3376 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34bffd1f0ca14092bc3cf0510b04b1bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34bffd1f0ca14092bc3cf0510b04b1bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40019 } }
I20260924 04:13:56.246434 3376 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34bffd1f0ca14092bc3cf0510b04b1bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:13:56.246501 3376 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34bffd1f0ca14092bc3cf0510b04b1bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:13:56.246598 3376 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34bffd1f0ca14092bc3cf0510b04b1bc [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:13:56.247422 3376 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34bffd1f0ca14092bc3cf0510b04b1bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34bffd1f0ca14092bc3cf0510b04b1bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40019 } }
I20260924 04:13:56.247723 3376 leader_election.cc:302] T 00000000000000000000000000000000 P 34bffd1f0ca14092bc3cf0510b04b1bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34bffd1f0ca14092bc3cf0510b04b1bc; no voters:
I20260924 04:13:56.248054 3376 leader_election.cc:288] T 00000000000000000000000000000000 P 34bffd1f0ca14092bc3cf0510b04b1bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:13:56.248164 3381 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34bffd1f0ca14092bc3cf0510b04b1bc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:13:56.248361 3381 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34bffd1f0ca14092bc3cf0510b04b1bc [term 1 LEADER]: Becoming Leader. State: Replica: 34bffd1f0ca14092bc3cf0510b04b1bc, State: Running, Role: LEADER
I20260924 04:13:56.248692 3381 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34bffd1f0ca14092bc3cf0510b04b1bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34bffd1f0ca14092bc3cf0510b04b1bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40019 } }
I20260924 04:13:56.249318 3376 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34bffd1f0ca14092bc3cf0510b04b1bc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:13:56.250308 3379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:56.238342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60110 (local address 127.0.98.62:40019)
0924 04:13:56.238721 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:56.238736 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:56.238802 (+ 66us) server_negotiation.cc:407] Connection header received
0924 04:13:56.239434 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:56.239452 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:56.239786 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:56.240110 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:56.240878 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:56.241835 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:56.242693 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:56.243163 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:56.243324 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:56.245562 (+ 2238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:13:56.245567 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:13:56.245580 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:13:56.245605 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:13:56.247299 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:56.247731 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:56.247737 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:56.247743 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:56.247878 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:56.248136 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:56.248139 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:56.248141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:56.248627 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:13:56.248822 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:56.249065 (+ 243us) server_negotiation.cc:299] Negotiation successful
0924 04:13:56.249349 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":104,"threads_started":1}
I20260924 04:13:56.250357 3383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34bffd1f0ca14092bc3cf0510b04b1bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34bffd1f0ca14092bc3cf0510b04b1bc. Latest consensus state: current_term: 1 leader_uuid: "34bffd1f0ca14092bc3cf0510b04b1bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34bffd1f0ca14092bc3cf0510b04b1bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40019 } } }
I20260924 04:13:56.250698 3383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34bffd1f0ca14092bc3cf0510b04b1bc [sys.catalog]: This master's current role is: LEADER
I20260924 04:13:56.250554 3382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34bffd1f0ca14092bc3cf0510b04b1bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34bffd1f0ca14092bc3cf0510b04b1bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34bffd1f0ca14092bc3cf0510b04b1bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40019 } } }
I20260924 04:13:56.250993 3382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34bffd1f0ca14092bc3cf0510b04b1bc [sys.catalog]: This master's current role is: LEADER
I20260924 04:13:56.253413 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260924 04:13:56.254732 3396 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 34bffd1f0ca14092bc3cf0510b04b1bc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:13:56.254846 3396 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:13:56.254957 3392 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:13:56.255798 3392 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:13:56.261410 3392 catalog_manager.cc:1347] Generated new cluster ID: ec80282403c84255801471f28cb56e04
I20260924 04:13:56.261475 3392 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:13:56.277912 3379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:56.272550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60122 (local address 127.0.98.62:40019)
0924 04:13:56.272654 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:56.272659 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:56.272857 (+ 198us) server_negotiation.cc:407] Connection header received
0924 04:13:56.272974 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:56.272978 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:56.273031 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:56.273117 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:56.273658 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:56.274445 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:56.275569 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:56.275933 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:56.276034 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:56.276880 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:13:56.276884 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:13:56.276887 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:13:56.277483 (+ 596us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:13:56.277518 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:13:56.277537 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:13:56.277619 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:56.277702 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 04:13:56.277764 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 04:13:56.325009 3392 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:13:56.325938 3392 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:13:56.333245 3392 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34bffd1f0ca14092bc3cf0510b04b1bc: Generated new TSK 0
I20260924 04:13:56.333916 3392 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:13:56.342931 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 3308
2026-09-24T04:13:56Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1345 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:13:56 dist-test-slave-p401 krb5kdc[3409](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:13:56 dist-test-slave-p401 krb5kdc[3409](info): set up 2 sockets
Sep 24 04:13:56 dist-test-slave-p401 krb5kdc[3409](info): commencing operation
krb5kdc: starting...
W20260924 04:13:57.291273 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/krb5kdc/test-user.keytab.
Sep 24 04:13:57 dist-test-slave-p401 krb5kdc[3409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:13:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:13:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:13:57.407162 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33701
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33701
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:13:57.514559 3422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:13:57.514849 3422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:13:57.514907 3422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:13:57.518997 3422 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:13:57.519073 3422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:13:57.519100 3422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:13:57.519120 3422 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:13:57.519140 3422 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:13:57.524940 3422 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33701
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:13:57.526015 3422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:13:57.527030 3422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:13:57.535414 3427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:13:57.535414 3428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:13:57.535408 3430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:13:57.535801 3422 server_base.cc:1034] running on GCE node
I20260924 04:13:57.536350 3422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:13:57.537460 3422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:13:57.538622 3422 hybrid_clock.cc:648] HybridClock initialized: now 1790223237538605 us; error 33 us; skew 500 ppm
Sep 24 04:13:57 dist-test-slave-p401 krb5kdc[3409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223237, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:13:57.541522 3422 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:13:57.542456 3422 webserver.cc:460] Webserver started at http://127.0.98.62:41673/ using document root <none> and password file <none>
I20260924 04:13:57.542999 3422 fs_manager.cc:359] Metadata directory not provided
I20260924 04:13:57.543066 3422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:13:57.543320 3422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:13:57.545595 3422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/master-0/data/instance:
uuid: "9007289fb6d3459782a36157ea55f474"
format_stamp: "Formatted at 2026-09-24 04:13:57 on dist-test-slave-p401"
server_key: "d21c9750db7644ab812b1c4f9a20b21d"
server_key_iv: "da2c6ead4263219576ec5f28326d28ba"
server_key_version: "encryptionkey@0"
I20260924 04:13:57.546175 3422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/master-0/wal/instance:
uuid: "9007289fb6d3459782a36157ea55f474"
format_stamp: "Formatted at 2026-09-24 04:13:57 on dist-test-slave-p401"
server_key: "d21c9750db7644ab812b1c4f9a20b21d"
server_key_iv: "da2c6ead4263219576ec5f28326d28ba"
server_key_version: "encryptionkey@0"
I20260924 04:13:57.550087 3422 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 04:13:57.552717 3437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:13:57.554039 3422 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:13:57.554162 3422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/master-0/wal
uuid: "9007289fb6d3459782a36157ea55f474"
format_stamp: "Formatted at 2026-09-24 04:13:57 on dist-test-slave-p401"
server_key: "d21c9750db7644ab812b1c4f9a20b21d"
server_key_iv: "da2c6ead4263219576ec5f28326d28ba"
server_key_version: "encryptionkey@0"
I20260924 04:13:57.554262 3422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:13:57.567067 3422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:13:57.569859 3422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:13:57.570029 3422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:13:57.576460 3422 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33701
I20260924 04:13:57.576694 3489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33701 every 8 connection(s)
I20260924 04:13:57.577404 3422 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:13:57.580138 3490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:13:57.583221 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 3422
I20260924 04:13:57.583376 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/master-0/wal/instance
I20260924 04:13:57.583658 392 external_mini_cluster.cc:1428] Setting key f836bd7af15c6e81ab013665b00a9837
I20260924 04:13:57.585903 3490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9007289fb6d3459782a36157ea55f474: Bootstrap starting.
I20260924 04:13:57.588286 3490 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9007289fb6d3459782a36157ea55f474: Neither blocks nor log segments found. Creating new log.
I20260924 04:13:57.589099 3490 log.cc:824] T 00000000000000000000000000000000 P 9007289fb6d3459782a36157ea55f474: Log is configured to *not* fsync() on all Append() calls
I20260924 04:13:57.591307 3490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9007289fb6d3459782a36157ea55f474: No bootstrap required, opened a new log
Sep 24 04:13:57 dist-test-slave-p401 krb5kdc[3409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:13:57.595157 3490 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9007289fb6d3459782a36157ea55f474 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9007289fb6d3459782a36157ea55f474" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33701 } }
I20260924 04:13:57.595451 3490 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9007289fb6d3459782a36157ea55f474 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:13:57.595530 3490 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9007289fb6d3459782a36157ea55f474 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9007289fb6d3459782a36157ea55f474, State: Initialized, Role: FOLLOWER
I20260924 04:13:57.595995 3490 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9007289fb6d3459782a36157ea55f474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9007289fb6d3459782a36157ea55f474" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33701 } }
I20260924 04:13:57.596153 3490 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9007289fb6d3459782a36157ea55f474 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:13:57.596220 3490 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9007289fb6d3459782a36157ea55f474 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:13:57.596309 3490 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9007289fb6d3459782a36157ea55f474 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:13:57.597364 3490 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9007289fb6d3459782a36157ea55f474 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9007289fb6d3459782a36157ea55f474" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33701 } }
I20260924 04:13:57.597728 3490 leader_election.cc:302] T 00000000000000000000000000000000 P 9007289fb6d3459782a36157ea55f474 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9007289fb6d3459782a36157ea55f474; no voters:
I20260924 04:13:57.598040 3490 leader_election.cc:288] T 00000000000000000000000000000000 P 9007289fb6d3459782a36157ea55f474 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:13:57.598126 3495 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9007289fb6d3459782a36157ea55f474 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:13:57.598323 3493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:57.585001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42976 (local address 127.0.98.62:33701)
0924 04:13:57.585382 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:57.585399 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:57.585434 (+ 35us) server_negotiation.cc:407] Connection header received
0924 04:13:57.586055 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:57.586074 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:57.586410 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:57.586684 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:57.587585 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:57.588662 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:57.589795 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:57.590423 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:57.590568 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:57.593101 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:13:57.593108 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:13:57.593124 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:13:57.593154 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:13:57.595075 (+ 1921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:57.595547 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:57.595553 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:57.595559 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:57.595699 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:57.596026 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:57.596029 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:57.596031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:57.596545 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:13:57.596737 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:57.597040 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 04:13:57.597345 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":98,"threads_started":1}
I20260924 04:13:57.598342 3495 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9007289fb6d3459782a36157ea55f474 [term 1 LEADER]: Becoming Leader. State: Replica: 9007289fb6d3459782a36157ea55f474, State: Running, Role: LEADER
I20260924 04:13:57.598875 3495 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9007289fb6d3459782a36157ea55f474 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9007289fb6d3459782a36157ea55f474" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33701 } }
I20260924 04:13:57.599315 3490 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9007289fb6d3459782a36157ea55f474 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:13:57.600466 3496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9007289fb6d3459782a36157ea55f474 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9007289fb6d3459782a36157ea55f474" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9007289fb6d3459782a36157ea55f474" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33701 } } }
I20260924 04:13:57.600575 3496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9007289fb6d3459782a36157ea55f474 [sys.catalog]: This master's current role is: LEADER
I20260924 04:13:57.600906 3497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9007289fb6d3459782a36157ea55f474 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9007289fb6d3459782a36157ea55f474. Latest consensus state: current_term: 1 leader_uuid: "9007289fb6d3459782a36157ea55f474" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9007289fb6d3459782a36157ea55f474" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33701 } } }
I20260924 04:13:57.601034 3497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9007289fb6d3459782a36157ea55f474 [sys.catalog]: This master's current role is: LEADER
W20260924 04:13:57.603928 3510 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9007289fb6d3459782a36157ea55f474: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:13:57.603992 3510 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:13:57.604061 3504 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:13:57.604692 3504 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:13:57.610124 3504 catalog_manager.cc:1347] Generated new cluster ID: e9221ee69cd9438ab4519e59c9efbb1c
I20260924 04:13:57.610174 3504 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:13:57.623173 3504 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:13:57.624092 3504 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:13:57.632784 3504 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9007289fb6d3459782a36157ea55f474: Generated new TSK 0
I20260924 04:13:57.633493 3504 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:13:57.662998 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33701
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:13:57.771179 3516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:13:57.771533 3516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:13:57.771579 3516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:13:57.775693 3516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:13:57.775768 3516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:13:57.775830 3516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:13:57.781332 3516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:13:57.782437 3516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:13:57.783514 3516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:13:57.793170 3521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:13:57.793179 3524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:13:57.793287 3516 server_base.cc:1034] running on GCE node
W20260924 04:13:57.793444 3522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:13:57.793828 3516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:13:57.794494 3516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:13:57.795650 3516 hybrid_clock.cc:648] HybridClock initialized: now 1790223237795629 us; error 30 us; skew 500 ppm
Sep 24 04:13:57 dist-test-slave-p401 krb5kdc[3409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223237, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:13:57.798306 3516 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:13:57.799222 3516 webserver.cc:460] Webserver started at http://127.0.98.1:42071/ using document root <none> and password file <none>
I20260924 04:13:57.799829 3516 fs_manager.cc:359] Metadata directory not provided
I20260924 04:13:57.799891 3516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:13:57.800098 3516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:13:57.802098 3516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-0/data/instance:
uuid: "936551ad08aa46e293b23f447f96e1a8"
format_stamp: "Formatted at 2026-09-24 04:13:57 on dist-test-slave-p401"
server_key: "1a3e765afe35a17d2960da9b44a4fa8b"
server_key_iv: "970a63a434de8d1cff541e4429d60190"
server_key_version: "encryptionkey@0"
I20260924 04:13:57.802628 3516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-0/wal/instance:
uuid: "936551ad08aa46e293b23f447f96e1a8"
format_stamp: "Formatted at 2026-09-24 04:13:57 on dist-test-slave-p401"
server_key: "1a3e765afe35a17d2960da9b44a4fa8b"
server_key_iv: "970a63a434de8d1cff541e4429d60190"
server_key_version: "encryptionkey@0"
I20260924 04:13:57.806178 3516 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 04:13:57.808501 3531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:13:57.809651 3516 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:13:57.809751 3516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-0/wal
uuid: "936551ad08aa46e293b23f447f96e1a8"
format_stamp: "Formatted at 2026-09-24 04:13:57 on dist-test-slave-p401"
server_key: "1a3e765afe35a17d2960da9b44a4fa8b"
server_key_iv: "970a63a434de8d1cff541e4429d60190"
server_key_version: "encryptionkey@0"
I20260924 04:13:57.809844 3516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:13:57.840518 3516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:13:57.843825 3516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:13:57.843997 3516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:13:57.844542 3516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:13:57.845322 3516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:13:57.845368 3516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:13:57.845408 3516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:13:57.845428 3516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:13:57.853369 3516 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42149
I20260924 04:13:57.853399 3644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42149 every 8 connection(s)
I20260924 04:13:57.854249 3516 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:13:57.858817 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 3516
I20260924 04:13:57.858925 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:13:57.859169 392 external_mini_cluster.cc:1428] Setting key 30145c70d41f8b57034af0b16e8ed0a1
Sep 24 04:13:57 dist-test-slave-p401 krb5kdc[3409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223237, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:13:57.868027 3493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:57.856034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55161 (local address 127.0.98.62:33701)
0924 04:13:57.856161 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:57.856165 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:57.856941 (+ 776us) server_negotiation.cc:407] Connection header received
0924 04:13:57.858361 (+ 1420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:57.858364 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:57.858419 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:57.858490 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:57.859711 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:57.860270 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:57.861360 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:57.861699 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:57.861803 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:57.864512 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:13:57.864516 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:13:57.864519 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:13:57.864547 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:13:57.865910 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:57.866494 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:57.866497 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:57.866498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:57.866568 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:57.866913 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:57.866915 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:57.866916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:57.867081 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:13:57.867150 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:57.867798 (+ 648us) server_negotiation.cc:299] Negotiation successful
0924 04:13:57.867908 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:13:57.869099 3647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:57.856380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33701 (local address 127.0.98.1:55161)
0924 04:13:57.856789 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:57.856840 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:57.858134 (+ 1294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:57.858665 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:57.858671 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:57.859054 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:13:57.859574 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:57.859589 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:57.860381 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:57.860384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:57.861243 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:57.861252 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:57.861861 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:57.861865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:57.862070 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:57.862620 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:13:57.862641 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:13:57.864341 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:13:57.866025 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:57.866031 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:57.866049 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:57.866407 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:57.866645 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:57.866649 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:57.866651 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:57.866832 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:57.867171 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:13:57.867178 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:13:57.867544 (+ 366us) client_negotiation.cc:769] Sending connection context
0924 04:13:57.867765 (+ 221us) client_negotiation.cc:240] Negotiation successful
0924 04:13:57.868119 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":100,"threads_started":1}
I20260924 04:13:57.870251 3645 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33701
I20260924 04:13:57.870528 3645 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:13:57.871091 3645 heartbeater.cc:511] Master 127.0.98.62:33701 requested a full tablet report, sending...
I20260924 04:13:57.872745 3454 ts_manager.cc:194] Registered new tserver with Master: 936551ad08aa46e293b23f447f96e1a8 (127.0.98.1:42149)
I20260924 04:13:57.873905 3454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55161
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:13:57.884161 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33701
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:13:57.991365 3650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:13:57.991644 3650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:13:57.991693 3650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:13:57.995721 3650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:13:57.995784 3650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:13:57.995863 3650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:13:58.001226 3650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:13:58.002238 3650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:13:58.003213 3650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:13:58.012884 3656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:13:58.012946 3650 server_base.cc:1034] running on GCE node
W20260924 04:13:58.012920 3658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:13:58.012887 3655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:13:58.013597 3650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:13:58.014277 3650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:13:58.015443 3650 hybrid_clock.cc:648] HybridClock initialized: now 1790223238015429 us; error 26 us; skew 500 ppm
Sep 24 04:13:58 dist-test-slave-p401 krb5kdc[3409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:13:58.018090 3650 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:13:58.018968 3650 webserver.cc:460] Webserver started at http://127.0.98.2:41845/ using document root <none> and password file <none>
I20260924 04:13:58.019555 3650 fs_manager.cc:359] Metadata directory not provided
I20260924 04:13:58.019624 3650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:13:58.019831 3650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:13:58.021821 3650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-1/data/instance:
uuid: "294d1214851f487088d08a19b63f3abd"
format_stamp: "Formatted at 2026-09-24 04:13:58 on dist-test-slave-p401"
server_key: "63d5b777f1c4bcf0385b82d71ef6710f"
server_key_iv: "9ab2584cd4bfdcab2ba73f7ee98e5319"
server_key_version: "encryptionkey@0"
I20260924 04:13:58.022343 3650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-1/wal/instance:
uuid: "294d1214851f487088d08a19b63f3abd"
format_stamp: "Formatted at 2026-09-24 04:13:58 on dist-test-slave-p401"
server_key: "63d5b777f1c4bcf0385b82d71ef6710f"
server_key_iv: "9ab2584cd4bfdcab2ba73f7ee98e5319"
server_key_version: "encryptionkey@0"
I20260924 04:13:58.025913 3650 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260924 04:13:58.028316 3665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:13:58.029532 3650 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 04:13:58.029647 3650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-1/wal
uuid: "294d1214851f487088d08a19b63f3abd"
format_stamp: "Formatted at 2026-09-24 04:13:58 on dist-test-slave-p401"
server_key: "63d5b777f1c4bcf0385b82d71ef6710f"
server_key_iv: "9ab2584cd4bfdcab2ba73f7ee98e5319"
server_key_version: "encryptionkey@0"
I20260924 04:13:58.029779 3650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:13:58.042250 3650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:13:58.045367 3650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:13:58.045542 3650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:13:58.046227 3650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:13:58.047221 3650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:13:58.047310 3650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:13:58.047369 3650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:13:58.047397 3650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:13:58.055652 3650 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34549
I20260924 04:13:58.055672 3778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34549 every 8 connection(s)
I20260924 04:13:58.056505 3650 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:13:58.059721 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 3650
I20260924 04:13:58.059849 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:13:58.060149 392 external_mini_cluster.cc:1428] Setting key 49ff9d5ddbee96da1271a8fd34dc5b25
Sep 24 04:13:58 dist-test-slave-p401 krb5kdc[3409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:13:58.070461 3493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:58.058221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52365 (local address 127.0.98.62:33701)
0924 04:13:58.058341 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:58.058346 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:58.059247 (+ 901us) server_negotiation.cc:407] Connection header received
0924 04:13:58.060812 (+ 1565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:58.060816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:58.060863 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:58.060930 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:58.062336 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:58.062855 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:58.064015 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:58.064334 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:58.064434 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:58.067004 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:13:58.067007 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:13:58.067009 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:13:58.067031 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:13:58.068569 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:58.069144 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:58.069147 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:58.069148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:58.069220 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:58.069516 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:58.069518 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:58.069520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:58.069681 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:13:58.069750 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:58.070236 (+ 486us) server_negotiation.cc:299] Negotiation successful
0924 04:13:58.070333 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 04:13:58.071267 3781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:58.058682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33701 (local address 127.0.98.2:52365)
0924 04:13:58.059093 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:58.059135 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:58.060632 (+ 1497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:58.061114 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:58.061123 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:58.061538 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:13:58.062183 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:58.062203 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:58.062978 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:58.062981 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:58.063904 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:58.063913 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:58.064454 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:58.064456 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:58.064584 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:58.065162 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:13:58.065177 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:13:58.066843 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:13:58.068686 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:58.068692 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:58.068717 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:58.069047 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:58.069305 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:58.069308 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:58.069309 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:58.069445 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:58.069780 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:13:58.069785 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:13:58.070029 (+ 244us) client_negotiation.cc:769] Sending connection context
0924 04:13:58.070187 (+ 158us) client_negotiation.cc:240] Negotiation successful
0924 04:13:58.070438 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":106,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 04:13:58.072384 3779 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33701
I20260924 04:13:58.072607 3779 heartbeater.cc:464] Registering TS with master...
I20260924 04:13:58.073109 3779 heartbeater.cc:511] Master 127.0.98.62:33701 requested a full tablet report, sending...
I20260924 04:13:58.074150 3454 ts_manager.cc:194] Registered new tserver with Master: 294d1214851f487088d08a19b63f3abd (127.0.98.2:34549)
I20260924 04:13:58.074939 3454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52365
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:13:58.086890 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33701
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:13:58.197250 3784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:13:58.197536 3784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:13:58.197599 3784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:13:58.201754 3784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:13:58.201822 3784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:13:58.201902 3784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:13:58.207197 3784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:13:58.208310 3784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:13:58.209502 3784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:13:58.219091 3789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:13:58.219094 3792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:13:58.219096 3790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:13:58.219383 3784 server_base.cc:1034] running on GCE node
I20260924 04:13:58.219820 3784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:13:58.220464 3784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:13:58.221626 3784 hybrid_clock.cc:648] HybridClock initialized: now 1790223238221594 us; error 45 us; skew 500 ppm
Sep 24 04:13:58 dist-test-slave-p401 krb5kdc[3409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:13:58.224313 3784 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:13:58.225178 3784 webserver.cc:460] Webserver started at http://127.0.98.3:46017/ using document root <none> and password file <none>
I20260924 04:13:58.225716 3784 fs_manager.cc:359] Metadata directory not provided
I20260924 04:13:58.225780 3784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:13:58.225986 3784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:13:58.228034 3784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-2/data/instance:
uuid: "f7c48720eea7463aaacde4567963bdac"
format_stamp: "Formatted at 2026-09-24 04:13:58 on dist-test-slave-p401"
server_key: "ad07ea9a707c7db23fb7315e6785b2c5"
server_key_iv: "c1decb267d3cb46e18989f1753cf33d6"
server_key_version: "encryptionkey@0"
I20260924 04:13:58.228547 3784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f7c48720eea7463aaacde4567963bdac"
format_stamp: "Formatted at 2026-09-24 04:13:58 on dist-test-slave-p401"
server_key: "ad07ea9a707c7db23fb7315e6785b2c5"
server_key_iv: "c1decb267d3cb46e18989f1753cf33d6"
server_key_version: "encryptionkey@0"
I20260924 04:13:58.232128 3784 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:13:58.234534 3799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:13:58.235780 3784 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:13:58.235893 3784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-2/wal
uuid: "f7c48720eea7463aaacde4567963bdac"
format_stamp: "Formatted at 2026-09-24 04:13:58 on dist-test-slave-p401"
server_key: "ad07ea9a707c7db23fb7315e6785b2c5"
server_key_iv: "c1decb267d3cb46e18989f1753cf33d6"
server_key_version: "encryptionkey@0"
I20260924 04:13:58.235986 3784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:13:58.250602 3784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:13:58.253718 3784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:13:58.253902 3784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:13:58.254482 3784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:13:58.255321 3784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:13:58.255368 3784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:13:58.255421 3784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:13:58.255448 3784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:13:58.264042 3784 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39153
I20260924 04:13:58.264063 3912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39153 every 8 connection(s)
I20260924 04:13:58.264971 3784 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:13:58.273191 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 3784
I20260924 04:13:58.273301 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790223216265354-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:13:58.273550 392 external_mini_cluster.cc:1428] Setting key 872dc0b05a565798159d1b744daf98ef
Sep 24 04:13:58 dist-test-slave-p401 krb5kdc[3409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:13:58.278617 3493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:58.266796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52475 (local address 127.0.98.62:33701)
0924 04:13:58.266931 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:58.266935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:58.267903 (+ 968us) server_negotiation.cc:407] Connection header received
0924 04:13:58.269304 (+ 1401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:58.269308 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:58.269365 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:58.269460 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:58.270682 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:58.271207 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:58.272389 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:58.272608 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:58.272695 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:58.275435 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:13:58.275438 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:13:58.275441 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:13:58.275471 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:13:58.276651 (+ 1180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:58.277143 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:58.277147 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:58.277149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:58.277227 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:58.277551 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:58.277554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:58.277556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:58.277741 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:13:58.277827 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:58.278329 (+ 502us) server_negotiation.cc:299] Negotiation successful
0924 04:13:58.278458 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 04:13:58.279516 3915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:58.267177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33701 (local address 127.0.98.3:52475)
0924 04:13:58.267734 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:58.267782 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:58.269080 (+ 1298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:58.269660 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:58.269667 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:58.270044 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:13:58.270536 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:58.270550 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:58.271348 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:58.271352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:58.272265 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:58.272274 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:58.272749 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:58.272753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:58.272924 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:58.273607 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:13:58.273626 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:13:58.275284 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:13:58.276760 (+ 1476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:58.276766 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:58.276785 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:58.277049 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:58.277310 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:58.277313 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:58.277315 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:58.277465 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:58.277834 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:13:58.277841 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:13:58.278096 (+ 255us) client_negotiation.cc:769] Sending connection context
0924 04:13:58.278268 (+ 172us) client_negotiation.cc:240] Negotiation successful
0924 04:13:58.278627 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":176,"threads_started":1}
I20260924 04:13:58.280606 3913 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33701
I20260924 04:13:58.280854 3913 heartbeater.cc:464] Registering TS with master...
I20260924 04:13:58.281348 3913 heartbeater.cc:511] Master 127.0.98.62:33701 requested a full tablet report, sending...
I20260924 04:13:58.282285 3454 ts_manager.cc:194] Registered new tserver with Master: f7c48720eea7463aaacde4567963bdac (127.0.98.3:39153)
I20260924 04:13:58.282831 3454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52475
I20260924 04:13:58.287302 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 04:13:58 dist-test-slave-p401 krb5kdc[3409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223238, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 24 04:13:58 dist-test-slave-p401 krb5kdc[3409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223238, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:13:58.311976 3493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:58.303387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42990 (local address 127.0.98.62:33701)
0924 04:13:58.303532 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:58.303537 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:58.303687 (+ 150us) server_negotiation.cc:407] Connection header received
0924 04:13:58.303844 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:58.303847 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:58.303902 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:58.303965 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:58.304664 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:58.305187 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:58.306124 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:58.306325 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:58.306392 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:58.309071 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:13:58.309074 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:13:58.309076 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:13:58.309103 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:13:58.310351 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:58.310806 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:58.310809 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:58.310810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:58.310882 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:58.311148 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:58.311149 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:58.311151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:58.311422 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:13:58.311531 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:58.311722 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 04:13:58.311815 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260924 04:13:58.312336 3454 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:42990
I20260924 04:13:58.313117 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 3516
I20260924 04:13:58.318675 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 3650
I20260924 04:13:58.323091 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 3784
I20260924 04:13:58.327590 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 3422
2026-09-24T04:13:58Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1986 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:13:58 dist-test-slave-p401 krb5kdc[3925](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:13:58 dist-test-slave-p401 krb5kdc[3925](info): set up 2 sockets
Sep 24 04:13:58 dist-test-slave-p401 krb5kdc[3925](info): commencing operation
krb5kdc: starting...
W20260924 04:13:59.275019 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/krb5kdc/test-user.keytab.
Sep 24 04:13:59 dist-test-slave-p401 krb5kdc[3925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:13:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:13:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:13:59.390128 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42639
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42639
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:13:59.495654 3938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:13:59.495914 3938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:13:59.495968 3938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:13:59.499924 3938 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:13:59.499977 3938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:13:59.500003 3938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:13:59.500025 3938 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:13:59.500047 3938 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:13:59.505143 3938 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:13:59.506085 3938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:13:59.507032 3938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:13:59.514866 3943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:13:59.514860 3944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:13:59.515007 3938 server_base.cc:1034] running on GCE node
W20260924 04:13:59.514861 3946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:13:59.515638 3938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:13:59.516677 3938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:13:59.517858 3938 hybrid_clock.cc:648] HybridClock initialized: now 1790223239517831 us; error 42 us; skew 500 ppm
Sep 24 04:13:59 dist-test-slave-p401 krb5kdc[3925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223239, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:13:59.520637 3938 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:13:59.521545 3938 webserver.cc:460] Webserver started at http://127.0.98.62:36379/ using document root <none> and password file <none>
I20260924 04:13:59.522113 3938 fs_manager.cc:359] Metadata directory not provided
I20260924 04:13:59.522179 3938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:13:59.522359 3938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:13:59.524389 3938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/master-0/data/instance:
uuid: "e9bffe8b26bc42c8b473aecf63756be6"
format_stamp: "Formatted at 2026-09-24 04:13:59 on dist-test-slave-p401"
server_key: "5b951e7ea50530ef271d3baceb3314fc"
server_key_iv: "6088013078255d24f0ff7c9988a42757"
server_key_version: "encryptionkey@0"
I20260924 04:13:59.524914 3938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/master-0/wal/instance:
uuid: "e9bffe8b26bc42c8b473aecf63756be6"
format_stamp: "Formatted at 2026-09-24 04:13:59 on dist-test-slave-p401"
server_key: "5b951e7ea50530ef271d3baceb3314fc"
server_key_iv: "6088013078255d24f0ff7c9988a42757"
server_key_version: "encryptionkey@0"
I20260924 04:13:59.528419 3938 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:13:59.530710 3953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:13:59.531978 3938 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:13:59.532095 3938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/master-0/wal
uuid: "e9bffe8b26bc42c8b473aecf63756be6"
format_stamp: "Formatted at 2026-09-24 04:13:59 on dist-test-slave-p401"
server_key: "5b951e7ea50530ef271d3baceb3314fc"
server_key_iv: "6088013078255d24f0ff7c9988a42757"
server_key_version: "encryptionkey@0"
I20260924 04:13:59.532187 3938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:13:59.551127 3938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:13:59.553697 3938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:13:59.553872 3938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:13:59.560250 3938 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42639
I20260924 04:13:59.560261 4005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42639 every 8 connection(s)
I20260924 04:13:59.561143 3938 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:13:59.563691 4006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:13:59.565788 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 3938
I20260924 04:13:59.565918 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/master-0/wal/instance
I20260924 04:13:59.566190 392 external_mini_cluster.cc:1428] Setting key 71bf34548f2f1ac50d371186c1193ed6
I20260924 04:13:59.569475 4006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9bffe8b26bc42c8b473aecf63756be6: Bootstrap starting.
I20260924 04:13:59.571827 4006 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9bffe8b26bc42c8b473aecf63756be6: Neither blocks nor log segments found. Creating new log.
I20260924 04:13:59.572578 4006 log.cc:824] T 00000000000000000000000000000000 P e9bffe8b26bc42c8b473aecf63756be6: Log is configured to *not* fsync() on all Append() calls
I20260924 04:13:59.574309 4006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9bffe8b26bc42c8b473aecf63756be6: No bootstrap required, opened a new log
Sep 24 04:13:59 dist-test-slave-p401 krb5kdc[3925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:13:59.577310 4006 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9bffe8b26bc42c8b473aecf63756be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9bffe8b26bc42c8b473aecf63756be6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42639 } }
I20260924 04:13:59.577581 4006 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9bffe8b26bc42c8b473aecf63756be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:13:59.577642 4006 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9bffe8b26bc42c8b473aecf63756be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9bffe8b26bc42c8b473aecf63756be6, State: Initialized, Role: FOLLOWER
I20260924 04:13:59.578071 4006 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9bffe8b26bc42c8b473aecf63756be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9bffe8b26bc42c8b473aecf63756be6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42639 } }
I20260924 04:13:59.578197 4006 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9bffe8b26bc42c8b473aecf63756be6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:13:59.578259 4006 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9bffe8b26bc42c8b473aecf63756be6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:13:59.578351 4006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9bffe8b26bc42c8b473aecf63756be6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:13:59.579299 4006 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9bffe8b26bc42c8b473aecf63756be6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9bffe8b26bc42c8b473aecf63756be6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42639 } }
I20260924 04:13:59.579620 4006 leader_election.cc:302] T 00000000000000000000000000000000 P e9bffe8b26bc42c8b473aecf63756be6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9bffe8b26bc42c8b473aecf63756be6; no voters:
I20260924 04:13:59.579908 4006 leader_election.cc:288] T 00000000000000000000000000000000 P e9bffe8b26bc42c8b473aecf63756be6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:13:59.579946 4009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:59.567502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56684 (local address 127.0.98.62:42639)
0924 04:13:59.567859 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:59.567874 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:59.567908 (+ 34us) server_negotiation.cc:407] Connection header received
0924 04:13:59.568575 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:59.568592 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:59.568923 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:59.569190 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:59.570031 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:59.571033 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:59.572169 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:59.572623 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:59.572770 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:59.575100 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:13:59.575105 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:13:59.575119 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:13:59.575144 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:13:59.577067 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:59.577528 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:59.577532 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:59.577536 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:59.577665 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:59.577943 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:59.577946 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:59.577948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:59.578358 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:13:59.578557 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:59.578822 (+ 265us) server_negotiation.cc:299] Negotiation successful
0924 04:13:59.579090 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":88,"threads_started":1}
I20260924 04:13:59.580009 4011 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9bffe8b26bc42c8b473aecf63756be6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:13:59.580497 4011 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9bffe8b26bc42c8b473aecf63756be6 [term 1 LEADER]: Becoming Leader. State: Replica: e9bffe8b26bc42c8b473aecf63756be6, State: Running, Role: LEADER
I20260924 04:13:59.580874 4011 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9bffe8b26bc42c8b473aecf63756be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9bffe8b26bc42c8b473aecf63756be6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42639 } }
I20260924 04:13:59.581112 4006 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9bffe8b26bc42c8b473aecf63756be6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:13:59.582454 4013 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9bffe8b26bc42c8b473aecf63756be6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9bffe8b26bc42c8b473aecf63756be6. Latest consensus state: current_term: 1 leader_uuid: "e9bffe8b26bc42c8b473aecf63756be6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9bffe8b26bc42c8b473aecf63756be6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42639 } } }
I20260924 04:13:59.582851 4013 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9bffe8b26bc42c8b473aecf63756be6 [sys.catalog]: This master's current role is: LEADER
I20260924 04:13:59.582751 4012 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9bffe8b26bc42c8b473aecf63756be6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9bffe8b26bc42c8b473aecf63756be6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9bffe8b26bc42c8b473aecf63756be6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42639 } } }
I20260924 04:13:59.583606 4012 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9bffe8b26bc42c8b473aecf63756be6 [sys.catalog]: This master's current role is: LEADER
W20260924 04:13:59.585822 4025 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e9bffe8b26bc42c8b473aecf63756be6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:13:59.585901 4025 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:13:59.586004 4026 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:13:59.586745 4026 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:13:59.591739 4026 catalog_manager.cc:1347] Generated new cluster ID: 1c735252913d43fa83ccecbb92acfe65
I20260924 04:13:59.591820 4026 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:13:59.605074 4026 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:13:59.606355 4026 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:13:59.614986 4026 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9bffe8b26bc42c8b473aecf63756be6: Generated new TSK 0
I20260924 04:13:59.615818 4026 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:13:59.649907 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42639
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:13:59.759095 4032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:13:59.759428 4032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:13:59.759497 4032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:13:59.763506 4032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:13:59.763571 4032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:13:59.763649 4032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:13:59.768939 4032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:13:59.770002 4032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:13:59.771014 4032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:13:59.780441 4040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:13:59.780443 4038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:13:59.780519 4032 server_base.cc:1034] running on GCE node
W20260924 04:13:59.780441 4037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:13:59.781116 4032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:13:59.781805 4032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:13:59.782975 4032 hybrid_clock.cc:648] HybridClock initialized: now 1790223239782959 us; error 42 us; skew 500 ppm
Sep 24 04:13:59 dist-test-slave-p401 krb5kdc[3925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223239, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:13:59.785725 4032 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:13:59.786558 4032 webserver.cc:460] Webserver started at http://127.0.98.1:38233/ using document root <none> and password file <none>
I20260924 04:13:59.787067 4032 fs_manager.cc:359] Metadata directory not provided
I20260924 04:13:59.787124 4032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:13:59.787338 4032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:13:59.789319 4032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/ts-0/data/instance:
uuid: "f2e294225b6940f4a877f7fc8cfdc549"
format_stamp: "Formatted at 2026-09-24 04:13:59 on dist-test-slave-p401"
server_key: "2a85c03437b87af393420157f505411a"
server_key_iv: "c7efd37a0b9bdc42635a5fdb3ed05ca9"
server_key_version: "encryptionkey@0"
I20260924 04:13:59.789830 4032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f2e294225b6940f4a877f7fc8cfdc549"
format_stamp: "Formatted at 2026-09-24 04:13:59 on dist-test-slave-p401"
server_key: "2a85c03437b87af393420157f505411a"
server_key_iv: "c7efd37a0b9bdc42635a5fdb3ed05ca9"
server_key_version: "encryptionkey@0"
I20260924 04:13:59.793421 4032 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:13:59.795840 4047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:13:59.797035 4032 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:13:59.797132 4032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/ts-0/wal
uuid: "f2e294225b6940f4a877f7fc8cfdc549"
format_stamp: "Formatted at 2026-09-24 04:13:59 on dist-test-slave-p401"
server_key: "2a85c03437b87af393420157f505411a"
server_key_iv: "c7efd37a0b9bdc42635a5fdb3ed05ca9"
server_key_version: "encryptionkey@0"
I20260924 04:13:59.797233 4032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:13:59.816635 4032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:13:59.819855 4032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:13:59.820026 4032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:13:59.820611 4032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:13:59.821416 4032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:13:59.821460 4032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:13:59.821514 4032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:13:59.821544 4032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:13:59.829613 4032 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41643
I20260924 04:13:59.829640 4160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41643 every 8 connection(s)
I20260924 04:13:59.830475 4032 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:13:59.836117 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 4032
I20260924 04:13:59.836252 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:13:59.836496 392 external_mini_cluster.cc:1428] Setting key 00afea1e1d9250d9b9682b7ddf2f6b30
Sep 24 04:13:59 dist-test-slave-p401 krb5kdc[3925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223239, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:13:59.844017 4009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:59.832329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52595 (local address 127.0.98.62:42639)
0924 04:13:59.832463 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:59.832466 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:59.833318 (+ 852us) server_negotiation.cc:407] Connection header received
0924 04:13:59.834716 (+ 1398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:59.834719 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:59.834771 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:59.834843 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:59.836111 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:59.836737 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:59.838019 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:59.838246 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:59.838328 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:59.840830 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:13:59.840834 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:13:59.840837 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:13:59.840861 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:13:59.842135 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:59.842597 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:59.842600 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:59.842601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:59.842672 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:59.842978 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:59.842981 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:59.842983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:59.843151 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:13:59.843255 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:59.843728 (+ 473us) server_negotiation.cc:299] Negotiation successful
0924 04:13:59.843851 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:13:59.844870 4163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:59.832732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42639 (local address 127.0.98.1:52595)
0924 04:13:59.833158 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:59.833199 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:59.834515 (+ 1316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:59.835055 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:59.835062 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:59.835450 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:13:59.835974 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:59.835989 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:59.836844 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:59.836848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:59.837913 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:59.837923 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:59.838373 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:59.838378 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:59.838544 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:59.839077 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:13:59.839099 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:13:59.840677 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:13:59.842238 (+ 1561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:59.842244 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:59.842260 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:59.842505 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:59.842748 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:59.842750 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:59.842752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:59.842894 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:59.843264 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:13:59.843271 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:13:59.843511 (+ 240us) client_negotiation.cc:769] Sending connection context
0924 04:13:59.843675 (+ 164us) client_negotiation.cc:240] Negotiation successful
0924 04:13:59.844005 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":107,"threads_started":1}
I20260924 04:13:59.846014 4161 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42639
I20260924 04:13:59.846249 4161 heartbeater.cc:464] Registering TS with master...
I20260924 04:13:59.846716 4161 heartbeater.cc:511] Master 127.0.98.62:42639 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:13:59.848196 3970 ts_manager.cc:194] Registered new tserver with Master: f2e294225b6940f4a877f7fc8cfdc549 (127.0.98.1:41643)
I20260924 04:13:59.849628 3970 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52595
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:13:59.862114 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42639
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:13:59.967180 4166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:13:59.967475 4166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:13:59.967517 4166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:13:59.971457 4166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:13:59.971508 4166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:13:59.971563 4166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:13:59.976559 4166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:13:59.977514 4166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:13:59.978461 4166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:13:59.987758 4171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:13:59.987776 4174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:13:59.987828 4166 server_base.cc:1034] running on GCE node
W20260924 04:13:59.987836 4172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:13:59.988396 4166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:13:59.989023 4166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:13:59.990212 4166 hybrid_clock.cc:648] HybridClock initialized: now 1790223239990177 us; error 32 us; skew 500 ppm
Sep 24 04:13:59 dist-test-slave-p401 krb5kdc[3925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223239, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:13:59.992847 4166 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:13:59.993703 4166 webserver.cc:460] Webserver started at http://127.0.98.2:37213/ using document root <none> and password file <none>
I20260924 04:13:59.994217 4166 fs_manager.cc:359] Metadata directory not provided
I20260924 04:13:59.994274 4166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:13:59.994434 4166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:13:59.996420 4166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/ts-1/data/instance:
uuid: "e99ec2b97ac447398bbab122ce864252"
format_stamp: "Formatted at 2026-09-24 04:13:59 on dist-test-slave-p401"
server_key: "4d922d07829bacea45f5b8c9a7b9de6e"
server_key_iv: "8a75d801f548aaab08f3dbc4aba2843c"
server_key_version: "encryptionkey@0"
I20260924 04:13:59.996901 4166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e99ec2b97ac447398bbab122ce864252"
format_stamp: "Formatted at 2026-09-24 04:13:59 on dist-test-slave-p401"
server_key: "4d922d07829bacea45f5b8c9a7b9de6e"
server_key_iv: "8a75d801f548aaab08f3dbc4aba2843c"
server_key_version: "encryptionkey@0"
I20260924 04:14:00.000505 4166 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:14:00.002825 4181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:00.004217 4166 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:14:00.004338 4166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/ts-1/wal
uuid: "e99ec2b97ac447398bbab122ce864252"
format_stamp: "Formatted at 2026-09-24 04:13:59 on dist-test-slave-p401"
server_key: "4d922d07829bacea45f5b8c9a7b9de6e"
server_key_iv: "8a75d801f548aaab08f3dbc4aba2843c"
server_key_version: "encryptionkey@0"
I20260924 04:14:00.004462 4166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:14:00.024196 4166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:14:00.027465 4166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:14:00.027642 4166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:14:00.028245 4166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:14:00.029070 4166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:14:00.029116 4166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:00.029153 4166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:14:00.029179 4166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:00.037724 4166 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36257
I20260924 04:14:00.037742 4294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36257 every 8 connection(s)
I20260924 04:14:00.038610 4166 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 04:14:00 dist-test-slave-p401 krb5kdc[3925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223239, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:14:00.047930 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 4166
I20260924 04:14:00.048046 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:14:00.048300 392 external_mini_cluster.cc:1428] Setting key 67b8072da8b186c06fdf92e38d93f444
I20260924 04:14:00.052114 4009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:00.040362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33599 (local address 127.0.98.62:42639)
0924 04:14:00.040481 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:00.040484 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:00.041329 (+ 845us) server_negotiation.cc:407] Connection header received
0924 04:14:00.042681 (+ 1352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:00.042685 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:00.042744 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:00.042814 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:00.044049 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:00.044561 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:00.045692 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:00.045913 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:00.046002 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:00.048435 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:00.048438 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:00.048441 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:00.048471 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:00.050060 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:00.050515 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:00.050518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:00.050520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:00.050615 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:00.050885 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:00.050888 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:00.050889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:00.051091 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:00.051187 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:00.051832 (+ 645us) server_negotiation.cc:299] Negotiation successful
0924 04:14:00.051958 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 04:14:00.053123 4297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:00.040756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42639 (local address 127.0.98.2:33599)
0924 04:14:00.041176 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:00.041219 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:00.042465 (+ 1246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:00.042997 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:00.043005 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:00.043414 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:00.043904 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:00.043917 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:00.044674 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:00.044677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:00.045568 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:00.045577 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:00.046050 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:00.046054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:00.046212 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:00.046758 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:00.046774 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:00.048275 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:14:00.050163 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:00.050167 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:00.050179 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:00.050427 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:00.050688 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:00.050691 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:00.050692 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:00.050814 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:00.051190 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:14:00.051197 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:14:00.051544 (+ 347us) client_negotiation.cc:769] Sending connection context
0924 04:14:00.051746 (+ 202us) client_negotiation.cc:240] Negotiation successful
0924 04:14:00.052107 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":102,"threads_started":1}
I20260924 04:14:00.054490 4295 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42639
I20260924 04:14:00.054924 4295 heartbeater.cc:464] Registering TS with master...
I20260924 04:14:00.055631 4295 heartbeater.cc:511] Master 127.0.98.62:42639 requested a full tablet report, sending...
I20260924 04:14:00.056720 3970 ts_manager.cc:194] Registered new tserver with Master: e99ec2b97ac447398bbab122ce864252 (127.0.98.2:36257)
I20260924 04:14:00.057298 3970 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33599
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:14:00.075017 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42639
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:14:00.185178 4300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:14:00.185438 4300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:14:00.185480 4300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:14:00.189517 4300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:14:00.189579 4300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:14:00.189635 4300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:14:00.195029 4300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:14:00.196097 4300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:14:00.197083 4300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:14:00.206442 4306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:00.206485 4305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:00.206642 4308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:00.206909 4300 server_base.cc:1034] running on GCE node
I20260924 04:14:00.207295 4300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:14:00.207914 4300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:14:00.209064 4300 hybrid_clock.cc:648] HybridClock initialized: now 1790223240209024 us; error 50 us; skew 500 ppm
Sep 24 04:14:00 dist-test-slave-p401 krb5kdc[3925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:14:00.211695 4300 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:14:00.212553 4300 webserver.cc:460] Webserver started at http://127.0.98.3:38625/ using document root <none> and password file <none>
I20260924 04:14:00.213063 4300 fs_manager.cc:359] Metadata directory not provided
I20260924 04:14:00.213120 4300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:14:00.213279 4300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:14:00.215296 4300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/ts-2/data/instance:
uuid: "4d199b405b804aa19706ee78b7e99168"
format_stamp: "Formatted at 2026-09-24 04:14:00 on dist-test-slave-p401"
server_key: "97318f415d6d7ea25df8bb8b33273182"
server_key_iv: "ed105d4f1197b008d27102b77e229bb6"
server_key_version: "encryptionkey@0"
I20260924 04:14:00.215806 4300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4d199b405b804aa19706ee78b7e99168"
format_stamp: "Formatted at 2026-09-24 04:14:00 on dist-test-slave-p401"
server_key: "97318f415d6d7ea25df8bb8b33273182"
server_key_iv: "ed105d4f1197b008d27102b77e229bb6"
server_key_version: "encryptionkey@0"
I20260924 04:14:00.219328 4300 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260924 04:14:00.221706 4315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:00.222911 4300 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:14:00.223013 4300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/ts-2/wal
uuid: "4d199b405b804aa19706ee78b7e99168"
format_stamp: "Formatted at 2026-09-24 04:14:00 on dist-test-slave-p401"
server_key: "97318f415d6d7ea25df8bb8b33273182"
server_key_iv: "ed105d4f1197b008d27102b77e229bb6"
server_key_version: "encryptionkey@0"
I20260924 04:14:00.223106 4300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:14:00.246060 4300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:14:00.249162 4300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:14:00.249336 4300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:14:00.249914 4300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:14:00.250754 4300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:14:00.250800 4300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:00.250854 4300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:14:00.250882 4300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:00.259171 4300 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34993
I20260924 04:14:00.259202 4428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34993 every 8 connection(s)
I20260924 04:14:00.260174 4300 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:14:00.260914 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 4300
I20260924 04:14:00.261019 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790223216265354-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:14:00.261260 392 external_mini_cluster.cc:1428] Setting key bd1ba56b7747548877d291a1190d1ba8
Sep 24 04:14:00 dist-test-slave-p401 krb5kdc[3925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:14:00.272585 4009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:00.261964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37233 (local address 127.0.98.62:42639)
0924 04:14:00.262087 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:00.262091 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:00.263091 (+ 1000us) server_negotiation.cc:407] Connection header received
0924 04:14:00.264511 (+ 1420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:00.264514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:00.264558 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:00.264620 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:00.265689 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:00.266208 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:00.267290 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:00.267501 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:00.267570 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:00.269784 (+ 2214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:00.269787 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:00.269789 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:00.269809 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:00.270931 (+ 1122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:00.271417 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:00.271420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:00.271421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:00.271492 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:00.271748 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:00.271750 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:00.271751 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:00.271917 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:00.271985 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:00.272381 (+ 396us) server_negotiation.cc:299] Negotiation successful
0924 04:14:00.272469 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 04:14:00.273362 4431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:00.262464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42639 (local address 127.0.98.3:37233)
0924 04:14:00.262951 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:00.263004 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:00.264321 (+ 1317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:00.264760 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:00.264767 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:00.265115 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:00.265584 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:00.265598 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:00.266297 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:00.266300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:00.267170 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:00.267178 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:00.267604 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:00.267607 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:00.267737 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:00.268310 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:00.268325 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:00.269654 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:14:00.271024 (+ 1370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:00.271029 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:00.271041 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:00.271322 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:00.271553 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:00.271555 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:00.271556 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:00.271682 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:00.271986 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:14:00.271991 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:14:00.272204 (+ 213us) client_negotiation.cc:769] Sending connection context
0924 04:14:00.272342 (+ 138us) client_negotiation.cc:240] Negotiation successful
0924 04:14:00.272581 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":109,"threads_started":1}
I20260924 04:14:00.274410 4429 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42639
I20260924 04:14:00.274631 4429 heartbeater.cc:464] Registering TS with master...
I20260924 04:14:00.275091 4429 heartbeater.cc:511] Master 127.0.98.62:42639 requested a full tablet report, sending...
I20260924 04:14:00.276034 3970 ts_manager.cc:194] Registered new tserver with Master: 4d199b405b804aa19706ee78b7e99168 (127.0.98.3:34993)
I20260924 04:14:00.276582 3970 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37233
I20260924 04:14:00.286247 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 04:14:00 dist-test-slave-p401 krb5kdc[3925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 04:14:00 dist-test-slave-p401 krb5kdc[3925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:14:00.309875 4436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:00.301260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47174 (local address 127.0.98.1:41643)
0924 04:14:00.301509 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:00.301523 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:00.301548 (+ 25us) server_negotiation.cc:407] Connection header received
0924 04:14:00.301646 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:00.301650 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:00.301888 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:00.302003 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:00.302600 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:00.303421 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:00.304165 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:00.304530 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:00.304603 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:00.306830 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:00.306839 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:00.306853 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:00.306885 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:00.308406 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:00.308809 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:00.308814 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:00.308816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:00.308896 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:00.309082 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:00.309084 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:00.309085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:00.309306 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:00.309486 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:00.309610 (+ 124us) server_negotiation.cc:299] Negotiation successful
0924 04:14:00.309726 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":65,"threads_started":1}
Sep 24 04:14:00 dist-test-slave-p401 krb5kdc[3925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:14:00.319286 4009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:00.312572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56698 (local address 127.0.98.62:42639)
0924 04:14:00.312703 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:00.312706 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:00.312741 (+ 35us) server_negotiation.cc:407] Connection header received
0924 04:14:00.312833 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:00.312836 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:00.312874 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:00.312936 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:00.313432 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:00.313921 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:00.314602 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:00.314802 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:00.314878 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:00.316860 (+ 1982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:00.316863 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:00.316865 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:00.316889 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:00.317960 (+ 1071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:00.318277 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:00.318279 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:00.318280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:00.318345 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:00.318526 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:00.318528 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:00.318529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:00.318712 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:00.318810 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:00.319022 (+ 212us) server_negotiation.cc:299] Negotiation successful
0924 04:14:00.319110 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260924 04:14:00.319810 3970 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56698
I20260924 04:14:00.320389 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 4032
I20260924 04:14:00.325286 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 4166
I20260924 04:14:00.329903 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 4300
I20260924 04:14:00.334478 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 3938
2026-09-24T04:14:00Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2007 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:14:00 dist-test-slave-p401 krb5kdc[4441](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:14:00 dist-test-slave-p401 krb5kdc[4441](info): set up 2 sockets
Sep 24 04:14:00 dist-test-slave-p401 krb5kdc[4441](info): commencing operation
krb5kdc: starting...
W20260924 04:14:01.291103 392 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/krb5kdc/test-user.keytab.
Sep 24 04:14:01 dist-test-slave-p401 krb5kdc[4441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223241, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:14:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:14:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:14:01.406666 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45793
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45793
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:14:01.514317 4454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:14:01.514580 4454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:14:01.514621 4454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:14:01.518616 4454 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:14:01.518684 4454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:14:01.518703 4454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:14:01.518718 4454 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:14:01.518733 4454 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:14:01.523958 4454 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45793
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:14:01.524976 4454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:14:01.525944 4454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:14:01.533846 4462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:01.533841 4459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:01.533874 4460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:01.533922 4454 server_base.cc:1034] running on GCE node
I20260924 04:14:01.534531 4454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:14:01.535558 4454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:14:01.536706 4454 hybrid_clock.cc:648] HybridClock initialized: now 1790223241536694 us; error 24 us; skew 500 ppm
Sep 24 04:14:01 dist-test-slave-p401 krb5kdc[4441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:14:01.545857 4454 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:14:01.546572 4454 fs_manager.cc:359] Metadata directory not provided
I20260924 04:14:01.546639 4454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:14:01.546832 4454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:14:01.548998 4454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/master-0/data/instance:
uuid: "619acc0e83274a8eb80759399c2fb0b3"
format_stamp: "Formatted at 2026-09-24 04:14:01 on dist-test-slave-p401"
server_key: "f1114c65582d42294f8b97eb7d5af913"
server_key_iv: "ef8cea195514fecb8a69014ece06f265"
server_key_version: "encryptionkey@0"
I20260924 04:14:01.549520 4454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/master-0/wal/instance:
uuid: "619acc0e83274a8eb80759399c2fb0b3"
format_stamp: "Formatted at 2026-09-24 04:14:01 on dist-test-slave-p401"
server_key: "f1114c65582d42294f8b97eb7d5af913"
server_key_iv: "ef8cea195514fecb8a69014ece06f265"
server_key_version: "encryptionkey@0"
I20260924 04:14:01.553202 4454 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:14:01.555619 4468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:01.556891 4454 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:14:01.557010 4454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/master-0/wal
uuid: "619acc0e83274a8eb80759399c2fb0b3"
format_stamp: "Formatted at 2026-09-24 04:14:01 on dist-test-slave-p401"
server_key: "f1114c65582d42294f8b97eb7d5af913"
server_key_iv: "ef8cea195514fecb8a69014ece06f265"
server_key_version: "encryptionkey@0"
I20260924 04:14:01.557108 4454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:14:01.571105 4454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:14:01.574098 4454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:14:01.574267 4454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:14:01.580782 4454 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45793
I20260924 04:14:01.581022 4520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45793 every 8 connection(s)
I20260924 04:14:01.581426 4454 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:14:01.582454 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 4454
I20260924 04:14:01.582567 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/master-0/wal/instance
I20260924 04:14:01.582824 392 external_mini_cluster.cc:1428] Setting key db3b664f7207680365a1bdc15770d339
I20260924 04:14:01.584410 4521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:14:01.590513 4521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 619acc0e83274a8eb80759399c2fb0b3: Bootstrap starting.
Sep 24 04:14:01 dist-test-slave-p401 krb5kdc[4441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223241, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:14:01.592922 4521 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 619acc0e83274a8eb80759399c2fb0b3: Neither blocks nor log segments found. Creating new log.
I20260924 04:14:01.593621 4521 log.cc:824] T 00000000000000000000000000000000 P 619acc0e83274a8eb80759399c2fb0b3: Log is configured to *not* fsync() on all Append() calls
I20260924 04:14:01.595772 4521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 619acc0e83274a8eb80759399c2fb0b3: No bootstrap required, opened a new log
I20260924 04:14:01.596949 4524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:01.584097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36678 (local address 127.0.98.62:45793)
0924 04:14:01.584485 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:01.584501 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:01.584546 (+ 45us) server_negotiation.cc:407] Connection header received
0924 04:14:01.585215 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:01.585233 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:01.585557 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:01.585839 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:01.586698 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:01.587792 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:01.588889 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:01.589328 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:01.589435 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:01.592070 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:01.592075 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:01.592088 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:01.592116 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:01.593790 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:01.594267 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:01.594271 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:01.594275 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:01.594399 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:01.594647 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:01.594650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:01.594652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:01.595180 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:01.595387 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:01.595746 (+ 359us) server_negotiation.cc:299] Negotiation successful
0924 04:14:01.596039 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":120,"threads_started":1}
I20260924 04:14:01.599277 4521 raft_consensus.cc:348] T 00000000000000000000000000000000 P 619acc0e83274a8eb80759399c2fb0b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "619acc0e83274a8eb80759399c2fb0b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45793 } }
I20260924 04:14:01.599535 4521 raft_consensus.cc:374] T 00000000000000000000000000000000 P 619acc0e83274a8eb80759399c2fb0b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:14:01.599608 4521 raft_consensus.cc:729] T 00000000000000000000000000000000 P 619acc0e83274a8eb80759399c2fb0b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 619acc0e83274a8eb80759399c2fb0b3, State: Initialized, Role: FOLLOWER
I20260924 04:14:01.600023 4521 consensus_queue.cc:260] T 00000000000000000000000000000000 P 619acc0e83274a8eb80759399c2fb0b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "619acc0e83274a8eb80759399c2fb0b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45793 } }
I20260924 04:14:01.600147 4521 raft_consensus.cc:388] T 00000000000000000000000000000000 P 619acc0e83274a8eb80759399c2fb0b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:14:01.600214 4521 raft_consensus.cc:482] T 00000000000000000000000000000000 P 619acc0e83274a8eb80759399c2fb0b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:14:01.600304 4521 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 619acc0e83274a8eb80759399c2fb0b3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:14:01.601148 4521 raft_consensus.cc:504] T 00000000000000000000000000000000 P 619acc0e83274a8eb80759399c2fb0b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "619acc0e83274a8eb80759399c2fb0b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45793 } }
I20260924 04:14:01.601459 4521 leader_election.cc:302] T 00000000000000000000000000000000 P 619acc0e83274a8eb80759399c2fb0b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 619acc0e83274a8eb80759399c2fb0b3; no voters:
I20260924 04:14:01.601732 4521 leader_election.cc:288] T 00000000000000000000000000000000 P 619acc0e83274a8eb80759399c2fb0b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:14:01.601835 4526 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 619acc0e83274a8eb80759399c2fb0b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:14:01.602031 4526 raft_consensus.cc:686] T 00000000000000000000000000000000 P 619acc0e83274a8eb80759399c2fb0b3 [term 1 LEADER]: Becoming Leader. State: Replica: 619acc0e83274a8eb80759399c2fb0b3, State: Running, Role: LEADER
I20260924 04:14:01.602342 4526 consensus_queue.cc:237] T 00000000000000000000000000000000 P 619acc0e83274a8eb80759399c2fb0b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "619acc0e83274a8eb80759399c2fb0b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45793 } }
I20260924 04:14:01.602820 4521 sys_catalog.cc:564] T 00000000000000000000000000000000 P 619acc0e83274a8eb80759399c2fb0b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:14:01.603900 4527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 619acc0e83274a8eb80759399c2fb0b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "619acc0e83274a8eb80759399c2fb0b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "619acc0e83274a8eb80759399c2fb0b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45793 } } }
I20260924 04:14:01.604014 4527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 619acc0e83274a8eb80759399c2fb0b3 [sys.catalog]: This master's current role is: LEADER
I20260924 04:14:01.604554 4535 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:14:01.604804 4528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 619acc0e83274a8eb80759399c2fb0b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 619acc0e83274a8eb80759399c2fb0b3. Latest consensus state: current_term: 1 leader_uuid: "619acc0e83274a8eb80759399c2fb0b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "619acc0e83274a8eb80759399c2fb0b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45793 } } }
I20260924 04:14:01.604954 4528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 619acc0e83274a8eb80759399c2fb0b3 [sys.catalog]: This master's current role is: LEADER
I20260924 04:14:01.607353 4535 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:14:01.612363 4535 catalog_manager.cc:1347] Generated new cluster ID: 433054bead1c4fd29e5fe9423971e09c
I20260924 04:14:01.612432 4535 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:14:01.627310 4535 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:14:01.628567 4535 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:14:01.638331 4535 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 619acc0e83274a8eb80759399c2fb0b3: Generated new TSK 0
I20260924 04:14:01.639072 4535 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:14:01.668411 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45793
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:14:01.776801 4547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:14:01.777083 4547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:14:01.777144 4547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:14:01.781172 4547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:14:01.781246 4547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:14:01.781340 4547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:14:01.786684 4547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:14:01.787809 4547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:14:01.788849 4547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:14:01.798053 4555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:01.798053 4552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:01.798223 4547 server_base.cc:1034] running on GCE node
W20260924 04:14:01.798054 4553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:01.798830 4547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:14:01.799489 4547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:14:01.800650 4547 hybrid_clock.cc:648] HybridClock initialized: now 1790223241800632 us; error 29 us; skew 500 ppm
Sep 24 04:14:01 dist-test-slave-p401 krb5kdc[4441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:14:01.803382 4547 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:14:01.804174 4547 fs_manager.cc:359] Metadata directory not provided
I20260924 04:14:01.804246 4547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:14:01.804419 4547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:14:01.806403 4547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/ts-0/data/instance:
uuid: "eedd1c6e0e734a299c48b29078b2a845"
format_stamp: "Formatted at 2026-09-24 04:14:01 on dist-test-slave-p401"
server_key: "6eaa54d12df6a4df2cd0cc0737eade14"
server_key_iv: "5083b827a724cbba9d170e57fb2b3d2b"
server_key_version: "encryptionkey@0"
I20260924 04:14:01.806917 4547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/ts-0/wal/instance:
uuid: "eedd1c6e0e734a299c48b29078b2a845"
format_stamp: "Formatted at 2026-09-24 04:14:01 on dist-test-slave-p401"
server_key: "6eaa54d12df6a4df2cd0cc0737eade14"
server_key_iv: "5083b827a724cbba9d170e57fb2b3d2b"
server_key_version: "encryptionkey@0"
I20260924 04:14:01.810371 4547 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 04:14:01.812678 4561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:01.813856 4547 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:14:01.813946 4547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/ts-0/wal
uuid: "eedd1c6e0e734a299c48b29078b2a845"
format_stamp: "Formatted at 2026-09-24 04:14:01 on dist-test-slave-p401"
server_key: "6eaa54d12df6a4df2cd0cc0737eade14"
server_key_iv: "5083b827a724cbba9d170e57fb2b3d2b"
server_key_version: "encryptionkey@0"
I20260924 04:14:01.814025 4547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:14:01.828747 4547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:14:01.831481 4547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:14:01.831657 4547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:14:01.832207 4547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:14:01.833004 4547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:14:01.833050 4547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:01.833091 4547 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:14:01.833120 4547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:01.841310 4547 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41347
I20260924 04:14:01.841327 4674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41347 every 8 connection(s)
I20260924 04:14:01.841866 4547 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:14:01.844635 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 4547
I20260924 04:14:01.844758 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:14:01.845050 392 external_mini_cluster.cc:1428] Setting key 44807efb07dc8ef506fae62d1dc0f43e
Sep 24 04:14:01 dist-test-slave-p401 krb5kdc[4441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:14:01.856335 4524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:01.843547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43665 (local address 127.0.98.62:45793)
0924 04:14:01.843666 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:01.843669 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:01.844558 (+ 889us) server_negotiation.cc:407] Connection header received
0924 04:14:01.846129 (+ 1571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:01.846133 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:01.846177 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:01.846279 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:01.847798 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:01.848358 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:01.849953 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:01.850189 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:01.850268 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:01.852979 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:01.852983 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:01.852985 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:01.853009 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:01.854345 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:01.854834 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:01.854837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:01.854838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:01.854909 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:01.855211 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:01.855214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:01.855216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:01.855426 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:01.855510 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:01.856029 (+ 519us) server_negotiation.cc:299] Negotiation successful
0924 04:14:01.856155 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:14:01.857262 4677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:01.843939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45793 (local address 127.0.98.1:43665)
0924 04:14:01.844378 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:01.844435 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:01.845935 (+ 1500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:01.846476 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:01.846485 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:01.846956 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:01.847646 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:01.847664 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:01.848485 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:01.848488 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:01.849809 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:01.849820 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:01.850333 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:01.850338 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:01.850542 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:01.851160 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:01.851182 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:01.852816 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:14:01.854449 (+ 1633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:01.854454 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:01.854472 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:01.854742 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:01.854987 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:01.854990 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:01.854992 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:01.855124 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:01.855519 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:14:01.855526 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:14:01.855805 (+ 279us) client_negotiation.cc:769] Sending connection context
0924 04:14:01.855973 (+ 168us) client_negotiation.cc:240] Negotiation successful
0924 04:14:01.856302 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":103,"threads_started":1}
I20260924 04:14:01.858537 4675 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45793
I20260924 04:14:01.858816 4675 heartbeater.cc:464] Registering TS with master...
I20260924 04:14:01.859402 4675 heartbeater.cc:511] Master 127.0.98.62:45793 requested a full tablet report, sending...
I20260924 04:14:01.860984 4485 ts_manager.cc:194] Registered new tserver with Master: eedd1c6e0e734a299c48b29078b2a845 (127.0.98.1:41347)
I20260924 04:14:01.862149 4485 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43665
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:14:01.873975 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45793
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:14:01.980909 4680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:14:01.981175 4680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:14:01.981218 4680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:14:01.985163 4680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:14:01.985219 4680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:14:01.985277 4680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:14:01.990494 4680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:14:01.991542 4680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:14:01.992540 4680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:14:02.001520 4686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:02.001490 4685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:02.001572 4680 server_base.cc:1034] running on GCE node
W20260924 04:14:02.001468 4688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:02.002055 4680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:14:02.002648 4680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:14:02.003793 4680 hybrid_clock.cc:648] HybridClock initialized: now 1790223242003780 us; error 23 us; skew 500 ppm
Sep 24 04:14:02 dist-test-slave-p401 krb5kdc[4441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223242, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:14:02.006338 4680 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:14:02.007033 4680 fs_manager.cc:359] Metadata directory not provided
I20260924 04:14:02.007102 4680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:14:02.007334 4680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:14:02.009299 4680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/ts-1/data/instance:
uuid: "a089a5c9a0e743aab3f1cd30efb497f7"
format_stamp: "Formatted at 2026-09-24 04:14:02 on dist-test-slave-p401"
server_key: "d8394e817092af4da47f299ba0868d98"
server_key_iv: "a38c30f3cc71b6441eaa84e8697a53ef"
server_key_version: "encryptionkey@0"
I20260924 04:14:02.009825 4680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a089a5c9a0e743aab3f1cd30efb497f7"
format_stamp: "Formatted at 2026-09-24 04:14:02 on dist-test-slave-p401"
server_key: "d8394e817092af4da47f299ba0868d98"
server_key_iv: "a38c30f3cc71b6441eaa84e8697a53ef"
server_key_version: "encryptionkey@0"
I20260924 04:14:02.013414 4680 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:14:02.015775 4694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:02.016984 4680 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:14:02.017088 4680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/ts-1/wal
uuid: "a089a5c9a0e743aab3f1cd30efb497f7"
format_stamp: "Formatted at 2026-09-24 04:14:02 on dist-test-slave-p401"
server_key: "d8394e817092af4da47f299ba0868d98"
server_key_iv: "a38c30f3cc71b6441eaa84e8697a53ef"
server_key_version: "encryptionkey@0"
I20260924 04:14:02.017181 4680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:14:02.027839 4680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:14:02.030658 4680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:14:02.030835 4680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:14:02.031420 4680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:14:02.032218 4680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:14:02.032265 4680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:02.032317 4680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:14:02.032346 4680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:02.040545 4680 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37193
I20260924 04:14:02.040566 4807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37193 every 8 connection(s)
I20260924 04:14:02.041107 4680 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:14:02.049899 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 4680
I20260924 04:14:02.050010 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/ts-1/wal/instance
Sep 24 04:14:02 dist-test-slave-p401 krb5kdc[4441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223242, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:14:02.050266 392 external_mini_cluster.cc:1428] Setting key f21364ab5ab885678e5503b18aaca7b2
I20260924 04:14:02.054239 4524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:02.042756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47373 (local address 127.0.98.62:45793)
0924 04:14:02.042886 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:02.042890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:02.043698 (+ 808us) server_negotiation.cc:407] Connection header received
0924 04:14:02.045028 (+ 1330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:02.045031 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:02.045086 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:02.045158 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:02.046295 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:02.046821 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:02.047957 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:02.048176 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:02.048263 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:02.050618 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:02.050621 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:02.050623 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:02.050647 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:02.052264 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:02.052713 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:02.052717 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:02.052719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:02.052811 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:02.053090 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:02.053093 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:02.053095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:02.053317 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:02.053391 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:02.054001 (+ 610us) server_negotiation.cc:299] Negotiation successful
0924 04:14:02.054098 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 04:14:02.055349 4810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:02.043112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45793 (local address 127.0.98.2:47373)
0924 04:14:02.043548 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:02.043589 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:02.044810 (+ 1221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:02.045337 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:02.045344 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:02.045674 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:02.046150 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:02.046163 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:02.046936 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:02.046938 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:02.047830 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:02.047838 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:02.048308 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:02.048312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:02.048470 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:02.049020 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:02.049034 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:02.050462 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:14:02.052375 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:02.052379 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:02.052392 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:02.052629 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:02.052885 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:02.052887 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:02.052888 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:02.053013 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:02.053415 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:14:02.053422 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:14:02.053750 (+ 328us) client_negotiation.cc:769] Sending connection context
0924 04:14:02.053962 (+ 212us) client_negotiation.cc:240] Negotiation successful
0924 04:14:02.054293 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":94,"threads_started":1}
I20260924 04:14:02.056543 4808 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45793
I20260924 04:14:02.056816 4808 heartbeater.cc:464] Registering TS with master...
I20260924 04:14:02.057252 4808 heartbeater.cc:511] Master 127.0.98.62:45793 requested a full tablet report, sending...
I20260924 04:14:02.058284 4485 ts_manager.cc:194] Registered new tserver with Master: a089a5c9a0e743aab3f1cd30efb497f7 (127.0.98.2:37193)
I20260924 04:14:02.059070 4485 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47373
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:14:02.077664 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45793
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:14:02.185256 4813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:14:02.185552 4813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:14:02.185617 4813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:14:02.189729 4813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:14:02.189800 4813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:14:02.189888 4813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:14:02.195356 4813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:14:02.196403 4813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:14:02.197697 4813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:14:02.207159 4821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:02.207194 4819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:02.207214 4818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:02.207288 4813 server_base.cc:1034] running on GCE node
I20260924 04:14:02.207854 4813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:14:02.208509 4813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:14:02.209676 4813 hybrid_clock.cc:648] HybridClock initialized: now 1790223242209644 us; error 41 us; skew 500 ppm
Sep 24 04:14:02 dist-test-slave-p401 krb5kdc[4441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223242, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:14:02.212395 4813 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:14:02.213146 4813 fs_manager.cc:359] Metadata directory not provided
I20260924 04:14:02.213218 4813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:14:02.213413 4813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:14:02.215770 4813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/ts-2/data/instance:
uuid: "cd79be4f7cda45ea94a5c742e140f2e0"
format_stamp: "Formatted at 2026-09-24 04:14:02 on dist-test-slave-p401"
server_key: "2536a43396816d9c40ac07206f1bccf5"
server_key_iv: "80339985ba4ae163232f11819c44f2bb"
server_key_version: "encryptionkey@0"
I20260924 04:14:02.216313 4813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/ts-2/wal/instance:
uuid: "cd79be4f7cda45ea94a5c742e140f2e0"
format_stamp: "Formatted at 2026-09-24 04:14:02 on dist-test-slave-p401"
server_key: "2536a43396816d9c40ac07206f1bccf5"
server_key_iv: "80339985ba4ae163232f11819c44f2bb"
server_key_version: "encryptionkey@0"
I20260924 04:14:02.220129 4813 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 04:14:02.222532 4827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:02.223862 4813 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260924 04:14:02.223963 4813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/ts-2/wal
uuid: "cd79be4f7cda45ea94a5c742e140f2e0"
format_stamp: "Formatted at 2026-09-24 04:14:02 on dist-test-slave-p401"
server_key: "2536a43396816d9c40ac07206f1bccf5"
server_key_iv: "80339985ba4ae163232f11819c44f2bb"
server_key_version: "encryptionkey@0"
I20260924 04:14:02.224061 4813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:14:02.237883 4813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:14:02.240873 4813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:14:02.241047 4813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:14:02.241647 4813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:14:02.242486 4813 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:14:02.242532 4813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:02.242573 4813 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:14:02.242602 4813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:02.251021 4813 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39723
I20260924 04:14:02.251039 4940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39723 every 8 connection(s)
I20260924 04:14:02.251679 4813 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:14:02.253263 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 4813
I20260924 04:14:02.253350 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:14:02.253630 392 external_mini_cluster.cc:1428] Setting key 0f1c8e19bcab47b66a862d0a4531e6df
Sep 24 04:14:02 dist-test-slave-p401 krb5kdc[4441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223242, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:14:02.265388 4524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:02.253429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45457 (local address 127.0.98.62:45793)
0924 04:14:02.253580 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:02.253585 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:02.254589 (+ 1004us) server_negotiation.cc:407] Connection header received
0924 04:14:02.256204 (+ 1615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:02.256207 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:02.256261 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:02.256331 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:02.257515 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:02.258036 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:02.259168 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:02.259433 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:02.259526 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:02.262027 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:02.262031 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:02.262033 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:02.262057 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:02.263465 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:02.263937 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:02.263941 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:02.263943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:02.264021 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:02.264325 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:02.264328 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:02.264330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:02.264511 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:02.264593 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:02.265098 (+ 505us) server_negotiation.cc:299] Negotiation successful
0924 04:14:02.265222 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 04:14:02.266233 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:02.253913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45793 (local address 127.0.98.3:45457)
0924 04:14:02.254442 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:02.254503 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:02.256009 (+ 1506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:02.256528 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:02.256535 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:02.256896 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:02.257380 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:02.257393 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:02.258149 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:02.258152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:02.259052 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:02.259061 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:02.259576 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:02.259580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:02.259756 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:02.260350 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:02.260379 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:02.261869 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:14:02.263570 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:02.263576 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:02.263593 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:02.263845 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:02.264100 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:02.264103 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:02.264105 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:02.264242 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:02.264611 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:14:02.264618 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:14:02.264872 (+ 254us) client_negotiation.cc:769] Sending connection context
0924 04:14:02.265043 (+ 171us) client_negotiation.cc:240] Negotiation successful
0924 04:14:02.265391 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":131,"threads_started":1}
I20260924 04:14:02.267416 4941 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45793
I20260924 04:14:02.267663 4941 heartbeater.cc:464] Registering TS with master...
I20260924 04:14:02.268098 4941 heartbeater.cc:511] Master 127.0.98.62:45793 requested a full tablet report, sending...
I20260924 04:14:02.269059 4485 ts_manager.cc:194] Registered new tserver with Master: cd79be4f7cda45ea94a5c742e140f2e0 (127.0.98.3:39723)
I20260924 04:14:02.269604 4485 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45457
I20260924 04:14:02.279552 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:14:02.288851 4524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:02.281398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36684 (local address 127.0.98.62:45793)
0924 04:14:02.281574 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:02.281578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:02.281705 (+ 127us) server_negotiation.cc:407] Connection header received
0924 04:14:02.281834 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:02.281838 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:02.281883 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:02.281949 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:02.282650 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:02.283202 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:02.284061 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:02.284278 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:02.284356 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:02.285737 (+ 1381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:02.285740 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:02.285743 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:02.285767 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:02.287059 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:02.287567 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:02.287580 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:02.287582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:02.287688 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:02.287959 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:02.287962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:02.287963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:02.288198 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:02.288310 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:02.288579 (+ 269us) server_negotiation.cc:299] Negotiation successful
0924 04:14:02.288688 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 04:14:02.291841 4485 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36684:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:14:02.294800 4485 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:14:02.309827 4953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:02.303636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37193 (local address 127.0.0.1:37080)
0924 04:14:02.304581 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:02.304594 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:02.304733 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:02.305682 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:02.305685 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:02.305702 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:14:02.305851 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:02.305858 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:02.307199 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:02.307202 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:02.308452 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:02.308462 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:02.309263 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:02.309267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:02.309379 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:02.309441 (+ 62us) client_negotiation.cc:769] Sending connection context
0924 04:14:02.309544 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 04:14:02.309672 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":853,"mutex_wait_us":10,"thread_start_us":125,"threads_started":1}
I20260924 04:14:02.309831 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:02.304492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37080 (local address 127.0.98.2:37193)
0924 04:14:02.305241 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:02.305253 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:02.305273 (+ 20us) server_negotiation.cc:407] Connection header received
0924 04:14:02.305336 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:02.305340 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:02.305534 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:02.305686 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:14:02.305968 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:02.307038 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:02.308596 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:02.309145 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:02.309247 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:02.309402 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:02.309586 (+ 184us) server_negotiation.cc:299] Negotiation successful
0924 04:14:02.309682 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":128,"threads_started":1}
I20260924 04:14:02.310163 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:02.303760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54450 (local address 127.0.98.1:41347)
0924 04:14:02.304811 (+ 1051us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:02.304824 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:02.304851 (+ 27us) server_negotiation.cc:407] Connection header received
0924 04:14:02.305018 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:02.305025 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:02.305244 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:02.305412 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:14:02.305810 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:02.307008 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:02.308589 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:02.309347 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:02.309448 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:02.309591 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:02.309875 (+ 284us) server_negotiation.cc:299] Negotiation successful
0924 04:14:02.309978 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":943,"thread_start_us":105,"threads_started":1}
I20260924 04:14:02.310211 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:02.303592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41347 (local address 127.0.0.1:54450)
0924 04:14:02.304248 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:02.304300 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:02.304839 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:02.305419 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:02.305425 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:02.305447 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:14:02.305683 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:02.305692 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:02.307154 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:02.307157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:02.308452 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:02.308462 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:02.309664 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:02.309667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:02.309731 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:02.309750 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 04:14:02.309825 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 04:14:02.310067 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"mutex_wait_us":145,"spinlock_wait_cycles":1792,"thread_start_us":100,"threads_started":1}
I20260924 04:14:02.310730 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:02.304483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52332 (local address 127.0.98.3:39723)
0924 04:14:02.305053 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:02.305066 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:02.305088 (+ 22us) server_negotiation.cc:407] Connection header received
0924 04:14:02.305169 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:02.305173 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:02.305314 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:02.305835 (+ 521us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:14:02.305997 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:02.306731 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:02.308341 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:02.309149 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:02.309443 (+ 294us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:02.309591 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:02.310317 (+ 726us) server_negotiation.cc:299] Negotiation successful
0924 04:14:02.310583 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":160,"threads_started":1}
I20260924 04:14:02.310254 4955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:02.303883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39723 (local address 127.0.0.1:52332)
0924 04:14:02.304936 (+ 1053us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:02.304947 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:02.305019 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:02.305465 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:02.305469 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:02.305487 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:14:02.305674 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:02.305682 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:02.306858 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:02.306862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:02.308199 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:02.308220 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:02.309305 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:02.309309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:02.309981 (+ 672us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:02.310003 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 04:14:02.310069 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 04:14:02.310137 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1000,"mutex_wait_us":653,"thread_start_us":79,"threads_started":1}
I20260924 04:14:02.313304 4742 tablet_service.cc:1467] Processing CreateTablet for tablet 5228f75f8f1c487a8e302ee3a585b867 (DEFAULT_TABLE table=test-table [id=b493000c5006425fbdb17d217183097b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:14:02.313402 4609 tablet_service.cc:1467] Processing CreateTablet for tablet 5228f75f8f1c487a8e302ee3a585b867 (DEFAULT_TABLE table=test-table [id=b493000c5006425fbdb17d217183097b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:14:02.314225 4742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5228f75f8f1c487a8e302ee3a585b867. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:14:02.313998 4875 tablet_service.cc:1467] Processing CreateTablet for tablet 5228f75f8f1c487a8e302ee3a585b867 (DEFAULT_TABLE table=test-table [id=b493000c5006425fbdb17d217183097b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:14:02.314440 4609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5228f75f8f1c487a8e302ee3a585b867. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:14:02.315023 4875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5228f75f8f1c487a8e302ee3a585b867. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:14:02.322257 4959 tablet_bootstrap.cc:492] T 5228f75f8f1c487a8e302ee3a585b867 P eedd1c6e0e734a299c48b29078b2a845: Bootstrap starting.
I20260924 04:14:02.323822 4958 tablet_bootstrap.cc:492] T 5228f75f8f1c487a8e302ee3a585b867 P a089a5c9a0e743aab3f1cd30efb497f7: Bootstrap starting.
I20260924 04:14:02.324179 4960 tablet_bootstrap.cc:492] T 5228f75f8f1c487a8e302ee3a585b867 P cd79be4f7cda45ea94a5c742e140f2e0: Bootstrap starting.
I20260924 04:14:02.324716 4959 tablet_bootstrap.cc:654] T 5228f75f8f1c487a8e302ee3a585b867 P eedd1c6e0e734a299c48b29078b2a845: Neither blocks nor log segments found. Creating new log.
I20260924 04:14:02.325812 4958 tablet_bootstrap.cc:654] T 5228f75f8f1c487a8e302ee3a585b867 P a089a5c9a0e743aab3f1cd30efb497f7: Neither blocks nor log segments found. Creating new log.
I20260924 04:14:02.326032 4959 log.cc:824] T 5228f75f8f1c487a8e302ee3a585b867 P eedd1c6e0e734a299c48b29078b2a845: Log is configured to *not* fsync() on all Append() calls
I20260924 04:14:02.326412 4960 tablet_bootstrap.cc:654] T 5228f75f8f1c487a8e302ee3a585b867 P cd79be4f7cda45ea94a5c742e140f2e0: Neither blocks nor log segments found. Creating new log.
I20260924 04:14:02.326954 4958 log.cc:824] T 5228f75f8f1c487a8e302ee3a585b867 P a089a5c9a0e743aab3f1cd30efb497f7: Log is configured to *not* fsync() on all Append() calls
I20260924 04:14:02.327633 4960 log.cc:824] T 5228f75f8f1c487a8e302ee3a585b867 P cd79be4f7cda45ea94a5c742e140f2e0: Log is configured to *not* fsync() on all Append() calls
I20260924 04:14:02.328553 4959 tablet_bootstrap.cc:492] T 5228f75f8f1c487a8e302ee3a585b867 P eedd1c6e0e734a299c48b29078b2a845: No bootstrap required, opened a new log
I20260924 04:14:02.328778 4959 ts_tablet_manager.cc:1397] T 5228f75f8f1c487a8e302ee3a585b867 P eedd1c6e0e734a299c48b29078b2a845: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260924 04:14:02.329071 4958 tablet_bootstrap.cc:492] T 5228f75f8f1c487a8e302ee3a585b867 P a089a5c9a0e743aab3f1cd30efb497f7: No bootstrap required, opened a new log
I20260924 04:14:02.329262 4958 ts_tablet_manager.cc:1397] T 5228f75f8f1c487a8e302ee3a585b867 P a089a5c9a0e743aab3f1cd30efb497f7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:14:02.329368 4960 tablet_bootstrap.cc:492] T 5228f75f8f1c487a8e302ee3a585b867 P cd79be4f7cda45ea94a5c742e140f2e0: No bootstrap required, opened a new log
I20260924 04:14:02.329538 4960 ts_tablet_manager.cc:1397] T 5228f75f8f1c487a8e302ee3a585b867 P cd79be4f7cda45ea94a5c742e140f2e0: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:14:02.332546 4960 raft_consensus.cc:348] T 5228f75f8f1c487a8e302ee3a585b867 P cd79be4f7cda45ea94a5c742e140f2e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eedd1c6e0e734a299c48b29078b2a845" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41347 } } peers { permanent_uuid: "a089a5c9a0e743aab3f1cd30efb497f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37193 } } peers { permanent_uuid: "cd79be4f7cda45ea94a5c742e140f2e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39723 } }
I20260924 04:14:02.332754 4960 raft_consensus.cc:374] T 5228f75f8f1c487a8e302ee3a585b867 P cd79be4f7cda45ea94a5c742e140f2e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:14:02.332787 4960 raft_consensus.cc:729] T 5228f75f8f1c487a8e302ee3a585b867 P cd79be4f7cda45ea94a5c742e140f2e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd79be4f7cda45ea94a5c742e140f2e0, State: Initialized, Role: FOLLOWER
I20260924 04:14:02.332922 4959 raft_consensus.cc:348] T 5228f75f8f1c487a8e302ee3a585b867 P eedd1c6e0e734a299c48b29078b2a845 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eedd1c6e0e734a299c48b29078b2a845" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41347 } } peers { permanent_uuid: "a089a5c9a0e743aab3f1cd30efb497f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37193 } } peers { permanent_uuid: "cd79be4f7cda45ea94a5c742e140f2e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39723 } }
I20260924 04:14:02.333212 4959 raft_consensus.cc:374] T 5228f75f8f1c487a8e302ee3a585b867 P eedd1c6e0e734a299c48b29078b2a845 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:14:02.333287 4959 raft_consensus.cc:729] T 5228f75f8f1c487a8e302ee3a585b867 P eedd1c6e0e734a299c48b29078b2a845 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eedd1c6e0e734a299c48b29078b2a845, State: Initialized, Role: FOLLOWER
I20260924 04:14:02.333204 4960 consensus_queue.cc:260] T 5228f75f8f1c487a8e302ee3a585b867 P cd79be4f7cda45ea94a5c742e140f2e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eedd1c6e0e734a299c48b29078b2a845" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41347 } } peers { permanent_uuid: "a089a5c9a0e743aab3f1cd30efb497f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37193 } } peers { permanent_uuid: "cd79be4f7cda45ea94a5c742e140f2e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39723 } }
I20260924 04:14:02.333357 4958 raft_consensus.cc:348] T 5228f75f8f1c487a8e302ee3a585b867 P a089a5c9a0e743aab3f1cd30efb497f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eedd1c6e0e734a299c48b29078b2a845" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41347 } } peers { permanent_uuid: "a089a5c9a0e743aab3f1cd30efb497f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37193 } } peers { permanent_uuid: "cd79be4f7cda45ea94a5c742e140f2e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39723 } }
I20260924 04:14:02.333649 4958 raft_consensus.cc:374] T 5228f75f8f1c487a8e302ee3a585b867 P a089a5c9a0e743aab3f1cd30efb497f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:14:02.333739 4958 raft_consensus.cc:729] T 5228f75f8f1c487a8e302ee3a585b867 P a089a5c9a0e743aab3f1cd30efb497f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a089a5c9a0e743aab3f1cd30efb497f7, State: Initialized, Role: FOLLOWER
I20260924 04:14:02.333767 4959 consensus_queue.cc:260] T 5228f75f8f1c487a8e302ee3a585b867 P eedd1c6e0e734a299c48b29078b2a845 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eedd1c6e0e734a299c48b29078b2a845" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41347 } } peers { permanent_uuid: "a089a5c9a0e743aab3f1cd30efb497f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37193 } } peers { permanent_uuid: "cd79be4f7cda45ea94a5c742e140f2e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39723 } }
I20260924 04:14:02.333899 4941 heartbeater.cc:503] Master 127.0.98.62:45793 was elected leader, sending a full tablet report...
I20260924 04:14:02.334280 4960 ts_tablet_manager.cc:1428] T 5228f75f8f1c487a8e302ee3a585b867 P cd79be4f7cda45ea94a5c742e140f2e0: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260924 04:14:02.334225 4958 consensus_queue.cc:260] T 5228f75f8f1c487a8e302ee3a585b867 P a089a5c9a0e743aab3f1cd30efb497f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eedd1c6e0e734a299c48b29078b2a845" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41347 } } peers { permanent_uuid: "a089a5c9a0e743aab3f1cd30efb497f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37193 } } peers { permanent_uuid: "cd79be4f7cda45ea94a5c742e140f2e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39723 } }
I20260924 04:14:02.334769 4959 ts_tablet_manager.cc:1428] T 5228f75f8f1c487a8e302ee3a585b867 P eedd1c6e0e734a299c48b29078b2a845: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260924 04:14:02.335088 4958 ts_tablet_manager.cc:1428] T 5228f75f8f1c487a8e302ee3a585b867 P a089a5c9a0e743aab3f1cd30efb497f7: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:14:02.335165 4675 heartbeater.cc:503] Master 127.0.98.62:45793 was elected leader, sending a full tablet report...
I20260924 04:14:02.335441 4808 heartbeater.cc:503] Master 127.0.98.62:45793 was elected leader, sending a full tablet report...
W20260924 04:14:02.342912 4676 tablet.cc:2369] T 5228f75f8f1c487a8e302ee3a585b867 P eedd1c6e0e734a299c48b29078b2a845: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:14:02.404922 4965 raft_consensus.cc:482] T 5228f75f8f1c487a8e302ee3a585b867 P eedd1c6e0e734a299c48b29078b2a845 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:14:02.405167 4965 raft_consensus.cc:504] T 5228f75f8f1c487a8e302ee3a585b867 P eedd1c6e0e734a299c48b29078b2a845 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eedd1c6e0e734a299c48b29078b2a845" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41347 } } peers { permanent_uuid: "a089a5c9a0e743aab3f1cd30efb497f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37193 } } peers { permanent_uuid: "cd79be4f7cda45ea94a5c742e140f2e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39723 } }
I20260924 04:14:02.406487 4965 leader_election.cc:288] T 5228f75f8f1c487a8e302ee3a585b867 P eedd1c6e0e734a299c48b29078b2a845 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a089a5c9a0e743aab3f1cd30efb497f7 (127.0.98.2:37193), cd79be4f7cda45ea94a5c742e140f2e0 (127.0.98.3:39723)
I20260924 04:14:02.411815 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:02.406828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51469 (local address 127.0.98.2:37193)
0924 04:14:02.407029 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:02.407033 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:02.407276 (+ 243us) server_negotiation.cc:407] Connection header received
0924 04:14:02.407330 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:02.407333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:02.407379 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:02.407462 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:14:02.407906 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:02.408697 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:02.410460 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:02.411091 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:02.411201 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:02.411299 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:02.411607 (+ 308us) server_negotiation.cc:299] Negotiation successful
0924 04:14:02.411680 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260924 04:14:02.411815 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:02.406899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41191 (local address 127.0.98.3:39723)
0924 04:14:02.407015 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:02.407020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:02.407111 (+ 91us) server_negotiation.cc:407] Connection header received
0924 04:14:02.407314 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:02.407318 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:02.407379 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:02.407462 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:14:02.407915 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:02.408696 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:02.410460 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:02.411099 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:02.411210 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:02.411299 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:02.411607 (+ 308us) server_negotiation.cc:299] Negotiation successful
0924 04:14:02.411680 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 04:14:02.411886 4967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:02.406724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39723 (local address 127.0.98.1:41191)
0924 04:14:02.407015 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:02.407031 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:02.407146 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:02.407510 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:02.407513 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:02.407557 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:14:02.407763 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:02.407771 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:02.408831 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:02.408835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:02.410328 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:02.410340 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:02.411267 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:02.411271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:02.411425 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:02.411509 (+ 84us) client_negotiation.cc:769] Sending connection context
0924 04:14:02.411583 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 04:14:02.411749 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":306,"spinlock_wait_cycles":29568,"thread_start_us":102,"threads_started":1}
I20260924 04:14:02.411885 4968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:02.406725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37193 (local address 127.0.98.1:51469)
0924 04:14:02.407138 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:02.407151 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:02.407253 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:02.407510 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:02.407513 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:02.407558 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:14:02.407763 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:02.407771 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:02.408831 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:02.408835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:02.410335 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:02.410341 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:02.411267 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:02.411271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:02.411456 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:02.411504 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 04:14:02.411583 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 04:14:02.411690 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"mutex_wait_us":171,"spinlock_wait_cycles":8448,"thread_start_us":124,"threads_started":1}
I20260924 04:14:02.412627 4895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5228f75f8f1c487a8e302ee3a585b867" candidate_uuid: "eedd1c6e0e734a299c48b29078b2a845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd79be4f7cda45ea94a5c742e140f2e0" is_pre_election: true
I20260924 04:14:02.412631 4762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5228f75f8f1c487a8e302ee3a585b867" candidate_uuid: "eedd1c6e0e734a299c48b29078b2a845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a089a5c9a0e743aab3f1cd30efb497f7" is_pre_election: true
I20260924 04:14:02.412914 4895 raft_consensus.cc:2454] T 5228f75f8f1c487a8e302ee3a585b867 P cd79be4f7cda45ea94a5c742e140f2e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eedd1c6e0e734a299c48b29078b2a845 in term 0.
I20260924 04:14:02.412914 4762 raft_consensus.cc:2454] T 5228f75f8f1c487a8e302ee3a585b867 P a089a5c9a0e743aab3f1cd30efb497f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eedd1c6e0e734a299c48b29078b2a845 in term 0.
I20260924 04:14:02.413370 4562 leader_election.cc:302] T 5228f75f8f1c487a8e302ee3a585b867 P eedd1c6e0e734a299c48b29078b2a845 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd79be4f7cda45ea94a5c742e140f2e0, eedd1c6e0e734a299c48b29078b2a845; no voters:
I20260924 04:14:02.413622 4965 raft_consensus.cc:2781] T 5228f75f8f1c487a8e302ee3a585b867 P eedd1c6e0e734a299c48b29078b2a845 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:14:02.413705 4965 raft_consensus.cc:482] T 5228f75f8f1c487a8e302ee3a585b867 P eedd1c6e0e734a299c48b29078b2a845 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:14:02.413753 4965 raft_consensus.cc:3037] T 5228f75f8f1c487a8e302ee3a585b867 P eedd1c6e0e734a299c48b29078b2a845 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:14:02.414759 4965 raft_consensus.cc:504] T 5228f75f8f1c487a8e302ee3a585b867 P eedd1c6e0e734a299c48b29078b2a845 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eedd1c6e0e734a299c48b29078b2a845" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41347 } } peers { permanent_uuid: "a089a5c9a0e743aab3f1cd30efb497f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37193 } } peers { permanent_uuid: "cd79be4f7cda45ea94a5c742e140f2e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39723 } }
I20260924 04:14:02.415086 4965 leader_election.cc:288] T 5228f75f8f1c487a8e302ee3a585b867 P eedd1c6e0e734a299c48b29078b2a845 [CANDIDATE]: Term 1 election: Requested vote from peers a089a5c9a0e743aab3f1cd30efb497f7 (127.0.98.2:37193), cd79be4f7cda45ea94a5c742e140f2e0 (127.0.98.3:39723)
I20260924 04:14:02.415494 4762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5228f75f8f1c487a8e302ee3a585b867" candidate_uuid: "eedd1c6e0e734a299c48b29078b2a845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a089a5c9a0e743aab3f1cd30efb497f7"
I20260924 04:14:02.415493 4895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5228f75f8f1c487a8e302ee3a585b867" candidate_uuid: "eedd1c6e0e734a299c48b29078b2a845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd79be4f7cda45ea94a5c742e140f2e0"
I20260924 04:14:02.415624 4895 raft_consensus.cc:3037] T 5228f75f8f1c487a8e302ee3a585b867 P cd79be4f7cda45ea94a5c742e140f2e0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:14:02.415624 4762 raft_consensus.cc:3037] T 5228f75f8f1c487a8e302ee3a585b867 P a089a5c9a0e743aab3f1cd30efb497f7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:14:02.416898 4762 raft_consensus.cc:2454] T 5228f75f8f1c487a8e302ee3a585b867 P a089a5c9a0e743aab3f1cd30efb497f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eedd1c6e0e734a299c48b29078b2a845 in term 1.
I20260924 04:14:02.416900 4895 raft_consensus.cc:2454] T 5228f75f8f1c487a8e302ee3a585b867 P cd79be4f7cda45ea94a5c742e140f2e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eedd1c6e0e734a299c48b29078b2a845 in term 1.
I20260924 04:14:02.417204 4565 leader_election.cc:302] T 5228f75f8f1c487a8e302ee3a585b867 P eedd1c6e0e734a299c48b29078b2a845 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a089a5c9a0e743aab3f1cd30efb497f7, eedd1c6e0e734a299c48b29078b2a845; no voters:
I20260924 04:14:02.417375 4965 raft_consensus.cc:2781] T 5228f75f8f1c487a8e302ee3a585b867 P eedd1c6e0e734a299c48b29078b2a845 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:14:02.417606 4965 raft_consensus.cc:686] T 5228f75f8f1c487a8e302ee3a585b867 P eedd1c6e0e734a299c48b29078b2a845 [term 1 LEADER]: Becoming Leader. State: Replica: eedd1c6e0e734a299c48b29078b2a845, State: Running, Role: LEADER
I20260924 04:14:02.417922 4965 consensus_queue.cc:237] T 5228f75f8f1c487a8e302ee3a585b867 P eedd1c6e0e734a299c48b29078b2a845 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eedd1c6e0e734a299c48b29078b2a845" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41347 } } peers { permanent_uuid: "a089a5c9a0e743aab3f1cd30efb497f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37193 } } peers { permanent_uuid: "cd79be4f7cda45ea94a5c742e140f2e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39723 } }
I20260924 04:14:02.421020 4485 catalog_manager.cc:5495] T 5228f75f8f1c487a8e302ee3a585b867 P eedd1c6e0e734a299c48b29078b2a845 reported cstate change: term changed from 0 to 1, leader changed from <none> to eedd1c6e0e734a299c48b29078b2a845 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "eedd1c6e0e734a299c48b29078b2a845" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eedd1c6e0e734a299c48b29078b2a845" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a089a5c9a0e743aab3f1cd30efb497f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd79be4f7cda45ea94a5c742e140f2e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39723 } health_report { overall_health: UNKNOWN } } }
I20260924 04:14:02.437237 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:02.434131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54462 (local address 127.0.98.1:41347)
0924 04:14:02.434244 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:02.434247 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:02.434278 (+ 31us) server_negotiation.cc:407] Connection header received
0924 04:14:02.434348 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:02.434350 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:02.434419 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:02.434498 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:14:02.434699 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:02.435183 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:02.435995 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:02.436202 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:02.436269 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:02.436527 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:14:02.436940 (+ 413us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:14:02.437024 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:02.437076 (+ 52us) server_negotiation.cc:299] Negotiation successful
0924 04:14:02.437126 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 04:14:02.443391 4895 raft_consensus.cc:1261] T 5228f75f8f1c487a8e302ee3a585b867 P cd79be4f7cda45ea94a5c742e140f2e0 [term 1 FOLLOWER]: Refusing update from remote peer eedd1c6e0e734a299c48b29078b2a845: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:14:02.443414 4762 raft_consensus.cc:1261] T 5228f75f8f1c487a8e302ee3a585b867 P a089a5c9a0e743aab3f1cd30efb497f7 [term 1 FOLLOWER]: Refusing update from remote peer eedd1c6e0e734a299c48b29078b2a845: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:14:02.444087 4965 consensus_queue.cc:1035] T 5228f75f8f1c487a8e302ee3a585b867 P eedd1c6e0e734a299c48b29078b2a845 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd79be4f7cda45ea94a5c742e140f2e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:14:02.444276 4969 consensus_queue.cc:1035] T 5228f75f8f1c487a8e302ee3a585b867 P eedd1c6e0e734a299c48b29078b2a845 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a089a5c9a0e743aab3f1cd30efb497f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:14:02.461382 4484 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:14:02.461572 4484 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:14:02.464010 4484 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 619acc0e83274a8eb80759399c2fb0b3: Sending DeleteTablet for 3 replicas of tablet 5228f75f8f1c487a8e302ee3a585b867
I20260924 04:14:02.464809 4609 tablet_service.cc:1514] Processing DeleteTablet for tablet 5228f75f8f1c487a8e302ee3a585b867 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:14:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54450
I20260924 04:14:02.464874 4742 tablet_service.cc:1514] Processing DeleteTablet for tablet 5228f75f8f1c487a8e302ee3a585b867 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:14:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37080
I20260924 04:14:02.464963 4875 tablet_service.cc:1514] Processing DeleteTablet for tablet 5228f75f8f1c487a8e302ee3a585b867 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:14:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52332
I20260924 04:14:02.465421 4984 tablet_replica.cc:330] T 5228f75f8f1c487a8e302ee3a585b867 P cd79be4f7cda45ea94a5c742e140f2e0: stopping tablet replica
I20260924 04:14:02.465588 4982 tablet_replica.cc:330] T 5228f75f8f1c487a8e302ee3a585b867 P eedd1c6e0e734a299c48b29078b2a845: stopping tablet replica
I20260924 04:14:02.465685 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 4547
I20260924 04:14:02.465900 4982 raft_consensus.cc:2229] T 5228f75f8f1c487a8e302ee3a585b867 P eedd1c6e0e734a299c48b29078b2a845 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:14:02.466573 4984 raft_consensus.cc:2229] T 5228f75f8f1c487a8e302ee3a585b867 P cd79be4f7cda45ea94a5c742e140f2e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:14:02.466653 4983 tablet_replica.cc:330] T 5228f75f8f1c487a8e302ee3a585b867 P a089a5c9a0e743aab3f1cd30efb497f7: stopping tablet replica
I20260924 04:14:02.466965 4983 raft_consensus.cc:2229] T 5228f75f8f1c487a8e302ee3a585b867 P a089a5c9a0e743aab3f1cd30efb497f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:14:02.467000 4984 raft_consensus.cc:2258] T 5228f75f8f1c487a8e302ee3a585b867 P cd79be4f7cda45ea94a5c742e140f2e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:14:02.467304 4983 raft_consensus.cc:2258] T 5228f75f8f1c487a8e302ee3a585b867 P a089a5c9a0e743aab3f1cd30efb497f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:14:02.468679 4983 ts_tablet_manager.cc:1905] T 5228f75f8f1c487a8e302ee3a585b867 P a089a5c9a0e743aab3f1cd30efb497f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:14:02.468695 4984 ts_tablet_manager.cc:1905] T 5228f75f8f1c487a8e302ee3a585b867 P cd79be4f7cda45ea94a5c742e140f2e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:14:02.470784 4983 ts_tablet_manager.cc:1918] T 5228f75f8f1c487a8e302ee3a585b867 P a089a5c9a0e743aab3f1cd30efb497f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:14:02.470871 4983 log.cc:1196] T 5228f75f8f1c487a8e302ee3a585b867 P a089a5c9a0e743aab3f1cd30efb497f7: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/ts-1/wal/wals/5228f75f8f1c487a8e302ee3a585b867
I20260924 04:14:02.471166 4983 ts_tablet_manager.cc:1939] T 5228f75f8f1c487a8e302ee3a585b867 P a089a5c9a0e743aab3f1cd30efb497f7: Deleting consensus metadata
I20260924 04:14:02.471165 4984 ts_tablet_manager.cc:1918] T 5228f75f8f1c487a8e302ee3a585b867 P cd79be4f7cda45ea94a5c742e140f2e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:14:02.471266 4984 log.cc:1196] T 5228f75f8f1c487a8e302ee3a585b867 P cd79be4f7cda45ea94a5c742e140f2e0: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790223216265354-392-0/minicluster-data/ts-2/wal/wals/5228f75f8f1c487a8e302ee3a585b867
I20260924 04:14:02.471608 4984 ts_tablet_manager.cc:1939] T 5228f75f8f1c487a8e302ee3a585b867 P cd79be4f7cda45ea94a5c742e140f2e0: Deleting consensus metadata
I20260924 04:14:02.472396 4472 catalog_manager.cc:4841] TS a089a5c9a0e743aab3f1cd30efb497f7 (127.0.98.2:37193): tablet 5228f75f8f1c487a8e302ee3a585b867 (table test-table [id=b493000c5006425fbdb17d217183097b]) successfully deleted
I20260924 04:14:02.472666 4469 catalog_manager.cc:4841] TS cd79be4f7cda45ea94a5c742e140f2e0 (127.0.98.3:39723): tablet 5228f75f8f1c487a8e302ee3a585b867 (table test-table [id=b493000c5006425fbdb17d217183097b]) successfully deleted
W20260924 04:14:02.473147 4470 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41347 (error 108)
I20260924 04:14:02.473338 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 4680
I20260924 04:14:02.474951 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:02.474664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41347 (local address 127.0.0.1:54478)
0924 04:14:02.474795 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:02.474866 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 04:14:02.475116 4470 catalog_manager.cc:4584] TS eedd1c6e0e734a299c48b29078b2a845 (127.0.98.1:41347): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5228f75f8f1c487a8e302ee3a585b867: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41347: connect: Connection refused (error 111)
I20260924 04:14:02.478310 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 4813
I20260924 04:14:02.482676 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 4454
2026-09-24T04:14:02Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2148 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-24T04:14:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:14:02Z Disabled control of system clock
I20260924 04:14:02.511344 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37891
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 04:14:02.619362 4989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:14:02.619652 4989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:14:02.619714 4989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:14:02.623731 4989 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:14:02.623812 4989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:14:02.623829 4989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:14:02.623848 4989 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:14:02.623865 4989 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:14:02.628794 4989 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37891
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:14:02.629853 4989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:14:02.631173 4989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:14:02.639756 4995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:02.639835 4989 server_base.cc:1034] running on GCE node
W20260924 04:14:02.639756 4997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:02.639750 4994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:02.640517 4989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:14:02.641604 4989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:14:02.642762 4989 hybrid_clock.cc:648] HybridClock initialized: now 1790223242642747 us; error 31 us; skew 500 ppm
I20260924 04:14:02.644626 4989 webserver.cc:460] Webserver started at http://127.0.98.62:36241/ using document root <none> and password file <none>
I20260924 04:14:02.645205 4989 fs_manager.cc:359] Metadata directory not provided
I20260924 04:14:02.645270 4989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:14:02.645464 4989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:14:02.647548 4989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/master-0/data/instance:
uuid: "93ff8b95282f4fdd8f6214e036efaf4e"
format_stamp: "Formatted at 2026-09-24 04:14:02 on dist-test-slave-p401"
server_key: "a5d2bddc52990ed25447cb3f9b2cb55f"
server_key_iv: "bd4f504f97e5e7caab5478b7eeb1fada"
server_key_version: "encryptionkey@0"
I20260924 04:14:02.648101 4989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/master-0/wal/instance:
uuid: "93ff8b95282f4fdd8f6214e036efaf4e"
format_stamp: "Formatted at 2026-09-24 04:14:02 on dist-test-slave-p401"
server_key: "a5d2bddc52990ed25447cb3f9b2cb55f"
server_key_iv: "bd4f504f97e5e7caab5478b7eeb1fada"
server_key_version: "encryptionkey@0"
I20260924 04:14:02.651861 4989 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260924 04:14:02.654225 5003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:02.655607 4989 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:14:02.655762 4989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/master-0/wal
uuid: "93ff8b95282f4fdd8f6214e036efaf4e"
format_stamp: "Formatted at 2026-09-24 04:14:02 on dist-test-slave-p401"
server_key: "a5d2bddc52990ed25447cb3f9b2cb55f"
server_key_iv: "bd4f504f97e5e7caab5478b7eeb1fada"
server_key_version: "encryptionkey@0"
I20260924 04:14:02.655864 4989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:14:02.660354 4989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:14:02.661079 4989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:14:02.661252 4989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:14:02.667937 4989 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37891
I20260924 04:14:02.667950 5055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37891 every 8 connection(s)
I20260924 04:14:02.668880 4989 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:14:02.672061 5056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:14:02.676545 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 4989
I20260924 04:14:02.676679 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/master-0/wal/instance
I20260924 04:14:02.676975 392 external_mini_cluster.cc:1428] Setting key 8ff897f678b324f87e6de115b1069f75
I20260924 04:14:02.677949 5056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93ff8b95282f4fdd8f6214e036efaf4e: Bootstrap starting.
I20260924 04:14:02.680231 5056 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93ff8b95282f4fdd8f6214e036efaf4e: Neither blocks nor log segments found. Creating new log.
I20260924 04:14:02.681365 5056 log.cc:824] T 00000000000000000000000000000000 P 93ff8b95282f4fdd8f6214e036efaf4e: Log is configured to *not* fsync() on all Append() calls
I20260924 04:14:02.682755 5059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:02.678475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47802 (local address 127.0.98.62:37891)
0924 04:14:02.679125 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:02.679139 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:02.679175 (+ 36us) server_negotiation.cc:407] Connection header received
0924 04:14:02.679842 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:02.679861 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:02.680233 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:02.680585 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:02.681297 (+ 712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:02.681303 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:02.681316 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:14:02.681325 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:02.681387 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:14:02.681403 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:02.681499 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:02.681639 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 04:14:02.682016 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":131,"threads_started":1}
I20260924 04:14:02.683441 5056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93ff8b95282f4fdd8f6214e036efaf4e: No bootstrap required, opened a new log
I20260924 04:14:02.686790 5056 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93ff8b95282f4fdd8f6214e036efaf4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93ff8b95282f4fdd8f6214e036efaf4e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37891 } }
I20260924 04:14:02.687002 5056 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93ff8b95282f4fdd8f6214e036efaf4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:14:02.687043 5056 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93ff8b95282f4fdd8f6214e036efaf4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93ff8b95282f4fdd8f6214e036efaf4e, State: Initialized, Role: FOLLOWER
I20260924 04:14:02.687482 5056 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93ff8b95282f4fdd8f6214e036efaf4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93ff8b95282f4fdd8f6214e036efaf4e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37891 } }
I20260924 04:14:02.687606 5056 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93ff8b95282f4fdd8f6214e036efaf4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:14:02.687665 5056 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93ff8b95282f4fdd8f6214e036efaf4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:14:02.687754 5056 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93ff8b95282f4fdd8f6214e036efaf4e [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:14:02.688614 5056 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93ff8b95282f4fdd8f6214e036efaf4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93ff8b95282f4fdd8f6214e036efaf4e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37891 } }
I20260924 04:14:02.688922 5056 leader_election.cc:302] T 00000000000000000000000000000000 P 93ff8b95282f4fdd8f6214e036efaf4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93ff8b95282f4fdd8f6214e036efaf4e; no voters:
I20260924 04:14:02.689203 5056 leader_election.cc:288] T 00000000000000000000000000000000 P 93ff8b95282f4fdd8f6214e036efaf4e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:14:02.689292 5061 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93ff8b95282f4fdd8f6214e036efaf4e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:14:02.689497 5061 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93ff8b95282f4fdd8f6214e036efaf4e [term 1 LEADER]: Becoming Leader. State: Replica: 93ff8b95282f4fdd8f6214e036efaf4e, State: Running, Role: LEADER
I20260924 04:14:02.689782 5061 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93ff8b95282f4fdd8f6214e036efaf4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93ff8b95282f4fdd8f6214e036efaf4e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37891 } }
I20260924 04:14:02.690268 5056 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93ff8b95282f4fdd8f6214e036efaf4e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:14:02.691421 5063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93ff8b95282f4fdd8f6214e036efaf4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93ff8b95282f4fdd8f6214e036efaf4e. Latest consensus state: current_term: 1 leader_uuid: "93ff8b95282f4fdd8f6214e036efaf4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93ff8b95282f4fdd8f6214e036efaf4e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37891 } } }
I20260924 04:14:02.691538 5063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93ff8b95282f4fdd8f6214e036efaf4e [sys.catalog]: This master's current role is: LEADER
I20260924 04:14:02.691787 5062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93ff8b95282f4fdd8f6214e036efaf4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93ff8b95282f4fdd8f6214e036efaf4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93ff8b95282f4fdd8f6214e036efaf4e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37891 } } }
I20260924 04:14:02.691870 5062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93ff8b95282f4fdd8f6214e036efaf4e [sys.catalog]: This master's current role is: LEADER
I20260924 04:14:02.691984 5070 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:14:02.694530 5070 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:14:02.699617 5070 catalog_manager.cc:1347] Generated new cluster ID: b2962336457044ebaea8bfdcd4e6dac5
I20260924 04:14:02.699673 5070 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:14:02.725371 5070 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:14:02.725845 5070 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:14:02.743470 5070 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93ff8b95282f4fdd8f6214e036efaf4e: Generated new TSK 0
I20260924 04:14:02.744360 5070 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:14:02.749205 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37891
--builtin_ntp_servers=127.0.98.20:35139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 04:14:02.859541 5080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:14:02.859853 5080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:14:02.859895 5080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:14:02.863960 5080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:14:02.864028 5080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:14:02.864086 5080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:14:02.868985 5080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:14:02.870033 5080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:14:02.871460 5080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:14:02.881031 5088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:02.881024 5085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:02.881032 5086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:02.881086 5080 server_base.cc:1034] running on GCE node
I20260924 04:14:02.881695 5080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:14:02.882373 5080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:14:02.883539 5080 hybrid_clock.cc:648] HybridClock initialized: now 1790223242883492 us; error 57 us; skew 500 ppm
I20260924 04:14:02.885462 5080 webserver.cc:460] Webserver started at http://127.0.98.1:41217/ using document root <none> and password file <none>
I20260924 04:14:02.886034 5080 fs_manager.cc:359] Metadata directory not provided
I20260924 04:14:02.886101 5080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:14:02.886305 5080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:14:02.888432 5080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/ts-0/data/instance:
uuid: "9d2ded6eb21b46f1990eddf60696d213"
format_stamp: "Formatted at 2026-09-24 04:14:02 on dist-test-slave-p401"
server_key: "abf49c30e999e82010edb833112f9bb2"
server_key_iv: "a67dfe04112b9ab1c9e5ba5763e6d590"
server_key_version: "encryptionkey@0"
I20260924 04:14:02.888980 5080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9d2ded6eb21b46f1990eddf60696d213"
format_stamp: "Formatted at 2026-09-24 04:14:02 on dist-test-slave-p401"
server_key: "abf49c30e999e82010edb833112f9bb2"
server_key_iv: "a67dfe04112b9ab1c9e5ba5763e6d590"
server_key_version: "encryptionkey@0"
I20260924 04:14:02.892872 5080 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:14:02.895326 5094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:02.896620 5080 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:14:02.896730 5080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/ts-0/wal
uuid: "9d2ded6eb21b46f1990eddf60696d213"
format_stamp: "Formatted at 2026-09-24 04:14:02 on dist-test-slave-p401"
server_key: "abf49c30e999e82010edb833112f9bb2"
server_key_iv: "a67dfe04112b9ab1c9e5ba5763e6d590"
server_key_version: "encryptionkey@0"
I20260924 04:14:02.896835 5080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:14:02.901367 5080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:14:02.901971 5080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:14:02.902128 5080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:14:02.902668 5080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:14:02.903626 5080 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:14:02.903672 5080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:02.903724 5080 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:14:02.903750 5080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:02.911816 5080 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34451
I20260924 04:14:02.911839 5207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34451 every 8 connection(s)
I20260924 04:14:02.912813 5080 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:14:02.915328 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 5080
I20260924 04:14:02.915452 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:14:02.915729 392 external_mini_cluster.cc:1428] Setting key 81deb61ac3b3c20a3ac792193b05b198
I20260924 04:14:02.917825 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37891
--builtin_ntp_servers=127.0.98.20:35139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 04:14:02.919140 5059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:02.914570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60743 (local address 127.0.98.62:37891)
0924 04:14:02.914694 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:02.914698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:02.915951 (+ 1253us) server_negotiation.cc:407] Connection header received
0924 04:14:02.917447 (+ 1496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:02.917450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:02.917484 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:02.917538 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:02.918718 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:02.918721 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:02.918723 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:14:02.918734 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:02.918755 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:14:02.918764 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:02.918832 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:02.918963 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 04:14:02.919006 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 04:14:02.920156 5210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:02.914982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37891 (local address 127.0.98.1:60743)
0924 04:14:02.915786 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:02.915841 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:02.917219 (+ 1378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:02.917735 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:02.917745 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:02.917868 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:02.918427 (+ 559us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:14:02.918429 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:02.918483 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:14:02.918573 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:14:02.918835 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:14:02.918841 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:14:02.918846 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 04:14:02.919012 (+ 166us) client_negotiation.cc:240] Negotiation successful
0924 04:14:02.919350 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":106,"threads_started":1}
I20260924 04:14:02.921286 5208 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37891
I20260924 04:14:02.921521 5208 heartbeater.cc:464] Registering TS with master...
I20260924 04:14:02.921918 5208 heartbeater.cc:511] Master 127.0.98.62:37891 requested a full tablet report, sending...
I20260924 04:14:02.923403 5020 ts_manager.cc:194] Registered new tserver with Master: 9d2ded6eb21b46f1990eddf60696d213 (127.0.98.1:34451)
W20260924 04:14:03.028204 5211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:14:03.028498 5211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:14:03.028554 5211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:14:03.032583 5211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:14:03.032647 5211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:14:03.032724 5211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:14:03.037596 5211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:14:03.038724 5211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:14:03.040131 5211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:14:03.049585 5219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:03.049695 5216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:03.049595 5217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:03.049772 5211 server_base.cc:1034] running on GCE node
I20260924 04:14:03.050252 5211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:14:03.050851 5211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:14:03.052011 5211 hybrid_clock.cc:648] HybridClock initialized: now 1790223243051984 us; error 36 us; skew 500 ppm
I20260924 04:14:03.053759 5211 webserver.cc:460] Webserver started at http://127.0.98.2:40123/ using document root <none> and password file <none>
I20260924 04:14:03.054299 5211 fs_manager.cc:359] Metadata directory not provided
I20260924 04:14:03.054366 5211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:14:03.054533 5211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:14:03.056586 5211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/ts-1/data/instance:
uuid: "a44efb30f23a4fcf930c4c03d4d9b5ff"
format_stamp: "Formatted at 2026-09-24 04:14:03 on dist-test-slave-p401"
server_key: "347ff1c02b882ae84ddb9fc84de90b90"
server_key_iv: "22634434c73d83b2a2a832870a67f04f"
server_key_version: "encryptionkey@0"
I20260924 04:14:03.057094 5211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a44efb30f23a4fcf930c4c03d4d9b5ff"
format_stamp: "Formatted at 2026-09-24 04:14:03 on dist-test-slave-p401"
server_key: "347ff1c02b882ae84ddb9fc84de90b90"
server_key_iv: "22634434c73d83b2a2a832870a67f04f"
server_key_version: "encryptionkey@0"
I20260924 04:14:03.060849 5211 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:14:03.063196 5225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:03.064500 5211 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:14:03.064620 5211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/ts-1/wal
uuid: "a44efb30f23a4fcf930c4c03d4d9b5ff"
format_stamp: "Formatted at 2026-09-24 04:14:03 on dist-test-slave-p401"
server_key: "347ff1c02b882ae84ddb9fc84de90b90"
server_key_iv: "22634434c73d83b2a2a832870a67f04f"
server_key_version: "encryptionkey@0"
I20260924 04:14:03.064736 5211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:14:03.070026 5211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:14:03.070601 5211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:14:03.070735 5211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:14:03.071249 5211 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:14:03.072151 5211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:14:03.072196 5211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:03.072232 5211 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:14:03.072252 5211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:03.080255 5211 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35753
I20260924 04:14:03.080305 5338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35753 every 8 connection(s)
I20260924 04:14:03.081100 5211 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:14:03.084028 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 5211
I20260924 04:14:03.084146 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:14:03.084412 392 external_mini_cluster.cc:1428] Setting key 1e55dbea01a200c267f1b5e267c321ba
I20260924 04:14:03.086341 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37891
--builtin_ntp_servers=127.0.98.20:35139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 04:14:03.087157 5059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:03.082776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35191 (local address 127.0.98.62:37891)
0924 04:14:03.082903 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:03.082906 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:03.083973 (+ 1067us) server_negotiation.cc:407] Connection header received
0924 04:14:03.085355 (+ 1382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:03.085358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:03.085399 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:03.085459 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:03.086723 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:03.086727 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:03.086730 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:14:03.086743 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:03.086767 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:14:03.086775 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:03.086855 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:03.086985 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 04:14:03.087029 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 04:14:03.088090 5341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:03.083166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37891 (local address 127.0.98.2:35191)
0924 04:14:03.083837 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:03.083876 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:03.085162 (+ 1286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:03.085653 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:03.085663 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:03.085790 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:03.086422 (+ 632us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:14:03.086425 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:03.086485 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:14:03.086578 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:14:03.086847 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:14:03.086854 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:14:03.086860 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 04:14:03.087025 (+ 165us) client_negotiation.cc:240] Negotiation successful
0924 04:14:03.087308 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":117,"threads_started":1}
I20260924 04:14:03.089221 5339 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37891
I20260924 04:14:03.089457 5339 heartbeater.cc:464] Registering TS with master...
I20260924 04:14:03.089859 5339 heartbeater.cc:511] Master 127.0.98.62:37891 requested a full tablet report, sending...
I20260924 04:14:03.090935 5020 ts_manager.cc:194] Registered new tserver with Master: a44efb30f23a4fcf930c4c03d4d9b5ff (127.0.98.2:35753)
W20260924 04:14:03.196923 5342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:14:03.197206 5342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:14:03.197261 5342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:14:03.201351 5342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:14:03.201435 5342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:14:03.201509 5342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:14:03.206297 5342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:14:03.207373 5342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:14:03.208714 5342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:14:03.218221 5350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:03.218221 5348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:03.218222 5347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:03.218537 5342 server_base.cc:1034] running on GCE node
I20260924 04:14:03.218997 5342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:14:03.219695 5342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:14:03.220873 5342 hybrid_clock.cc:648] HybridClock initialized: now 1790223243220859 us; error 26 us; skew 500 ppm
I20260924 04:14:03.222719 5342 webserver.cc:460] Webserver started at http://127.0.98.3:42829/ using document root <none> and password file <none>
I20260924 04:14:03.223320 5342 fs_manager.cc:359] Metadata directory not provided
I20260924 04:14:03.223384 5342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:14:03.223579 5342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:14:03.225594 5342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/ts-2/data/instance:
uuid: "83d680b6d463476bb20641ec909cd834"
format_stamp: "Formatted at 2026-09-24 04:14:03 on dist-test-slave-p401"
server_key: "de44125bd348226334faf098eddaf0db"
server_key_iv: "8c6c243d07d1a91bac7afae3615b26c7"
server_key_version: "encryptionkey@0"
I20260924 04:14:03.226111 5342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/ts-2/wal/instance:
uuid: "83d680b6d463476bb20641ec909cd834"
format_stamp: "Formatted at 2026-09-24 04:14:03 on dist-test-slave-p401"
server_key: "de44125bd348226334faf098eddaf0db"
server_key_iv: "8c6c243d07d1a91bac7afae3615b26c7"
server_key_version: "encryptionkey@0"
I20260924 04:14:03.229652 5342 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:14:03.232028 5356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:03.233287 5342 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:14:03.233388 5342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/ts-2/wal
uuid: "83d680b6d463476bb20641ec909cd834"
format_stamp: "Formatted at 2026-09-24 04:14:03 on dist-test-slave-p401"
server_key: "de44125bd348226334faf098eddaf0db"
server_key_iv: "8c6c243d07d1a91bac7afae3615b26c7"
server_key_version: "encryptionkey@0"
I20260924 04:14:03.233484 5342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:14:03.238083 5342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:14:03.238682 5342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:14:03.238837 5342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:14:03.239410 5342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:14:03.240304 5342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:14:03.240350 5342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:03.240401 5342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:14:03.240429 5342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:03.248982 5342 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39035
I20260924 04:14:03.249001 5469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39035 every 8 connection(s)
I20260924 04:14:03.249923 5342 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:14:03.252359 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 5342
I20260924 04:14:03.252463 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:14:03.252705 392 external_mini_cluster.cc:1428] Setting key f46e3871f96208491ed0dab2c7f0daf1
I20260924 04:14:03.256073 5059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:03.251732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55605 (local address 127.0.98.62:37891)
0924 04:14:03.251849 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:03.251853 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:03.253013 (+ 1160us) server_negotiation.cc:407] Connection header received
0924 04:14:03.254407 (+ 1394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:03.254410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:03.254451 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:03.254520 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:03.255606 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:03.255610 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:03.255613 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:14:03.255627 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:03.255652 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:14:03.255661 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:03.255739 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:03.255875 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 04:14:03.255921 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 04:14:03.256995 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:03.252114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37891 (local address 127.0.98.3:55605)
0924 04:14:03.252858 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:03.252910 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:03.254212 (+ 1302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:03.254684 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:03.254691 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:03.254783 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:03.255318 (+ 535us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:14:03.255320 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:03.255363 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:14:03.255448 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:14:03.255735 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:14:03.255743 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:14:03.255749 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 04:14:03.255915 (+ 166us) client_negotiation.cc:240] Negotiation successful
0924 04:14:03.256180 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":105,"threads_started":1}
I20260924 04:14:03.258145 5470 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37891
I20260924 04:14:03.258366 5470 heartbeater.cc:464] Registering TS with master...
I20260924 04:14:03.258769 5470 heartbeater.cc:511] Master 127.0.98.62:37891 requested a full tablet report, sending...
I20260924 04:14:03.259794 5020 ts_manager.cc:194] Registered new tserver with Master: 83d680b6d463476bb20641ec909cd834 (127.0.98.3:39035)
I20260924 04:14:03.266597 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:14:03.270274 5059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:03.268693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47814 (local address 127.0.98.62:37891)
0924 04:14:03.268866 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:03.268871 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:03.269016 (+ 145us) server_negotiation.cc:407] Connection header received
0924 04:14:03.269106 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:03.269108 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:03.269143 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:03.269231 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:03.269946 (+ 715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:03.269949 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:03.269951 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:14:03.269962 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:03.269981 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:14:03.269989 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:03.270062 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:03.270108 (+ 46us) server_negotiation.cc:299] Negotiation successful
0924 04:14:03.270139 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260924 04:14:03.273110 5020 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47814:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:14:03.276142 5020 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:14:03.287274 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:03.284156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34451 (local address 127.0.0.1:36250)
0924 04:14:03.284797 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:03.284848 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:03.284947 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:03.285431 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:03.285437 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:03.285519 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:03.286061 (+ 542us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:14:03.286063 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:03.286082 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:14:03.286139 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:14:03.286908 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:14:03.286916 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:14:03.286921 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 04:14:03.287035 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 04:14:03.287104 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":96,"threads_started":1}
I20260924 04:14:03.287350 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:03.284280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36250 (local address 127.0.98.1:34451)
0924 04:14:03.284915 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:03.284926 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:03.284951 (+ 25us) server_negotiation.cc:407] Connection header received
0924 04:14:03.285034 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:03.285039 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:03.285263 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:03.285789 (+ 526us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:03.286725 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:03.286731 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:03.286744 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:14:03.286754 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:03.286794 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:14:03.286830 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:03.286933 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:03.287063 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 04:14:03.287142 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":125,"threads_started":1}
I20260924 04:14:03.287595 5485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:03.284866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35753 (local address 127.0.0.1:54408)
0924 04:14:03.285621 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:03.285633 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:03.285695 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:03.286257 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:03.286260 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:03.286274 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:03.286722 (+ 448us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:14:03.286724 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:03.286734 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:14:03.286757 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:14:03.287359 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:14:03.287362 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:14:03.287365 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 04:14:03.287435 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 04:14:03.287467 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"thread_start_us":81,"threads_started":1}
I20260924 04:14:03.287961 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:03.285248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54408 (local address 127.0.98.2:35753)
0924 04:14:03.285897 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:03.285907 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:03.285928 (+ 21us) server_negotiation.cc:407] Connection header received
0924 04:14:03.285977 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:03.285982 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:03.286135 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:03.286262 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:03.287111 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:03.287117 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:03.287127 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:14:03.287138 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:03.287177 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:14:03.287207 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:03.287318 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:03.287754 (+ 436us) server_negotiation.cc:299] Negotiation successful
0924 04:14:03.287822 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":134,"threads_started":1}
I20260924 04:14:03.288221 5483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:03.284376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39035 (local address 127.0.0.1:47832)
0924 04:14:03.285429 (+ 1053us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:03.285446 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:03.285527 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:03.286655 (+ 1128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:03.286658 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:03.286672 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:03.287059 (+ 387us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:14:03.287060 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:03.287070 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:14:03.287093 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:14:03.288023 (+ 930us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:14:03.288025 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:14:03.288029 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 04:14:03.288096 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 04:14:03.288124 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":973,"thread_start_us":84,"threads_started":1}
I20260924 04:14:03.289173 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:03.284542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47832 (local address 127.0.98.3:39035)
0924 04:14:03.285191 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:03.285201 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:03.285618 (+ 417us) server_negotiation.cc:407] Connection header received
0924 04:14:03.285678 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:03.285683 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:03.285894 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:03.286020 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:03.287267 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:03.287274 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:03.287285 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:14:03.287298 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:03.287895 (+ 597us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:14:03.287933 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:03.288952 (+ 1019us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:03.288990 (+ 38us) server_negotiation.cc:299] Negotiation successful
0924 04:14:03.289046 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":78,"threads_started":1}
I20260924 04:14:03.290596 5142 tablet_service.cc:1467] Processing CreateTablet for tablet e302fa5313724aad96946d3d105f1690 (DEFAULT_TABLE table=test-table [id=69f5532a111e45978f8c42ee78bf79b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:14:03.290596 5273 tablet_service.cc:1467] Processing CreateTablet for tablet e302fa5313724aad96946d3d105f1690 (DEFAULT_TABLE table=test-table [id=69f5532a111e45978f8c42ee78bf79b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:14:03.291776 5142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e302fa5313724aad96946d3d105f1690. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:14:03.291776 5273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e302fa5313724aad96946d3d105f1690. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:14:03.291559 5404 tablet_service.cc:1467] Processing CreateTablet for tablet e302fa5313724aad96946d3d105f1690 (DEFAULT_TABLE table=test-table [id=69f5532a111e45978f8c42ee78bf79b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:14:03.292557 5404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e302fa5313724aad96946d3d105f1690. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:14:03.298384 5487 tablet_bootstrap.cc:492] T e302fa5313724aad96946d3d105f1690 P 83d680b6d463476bb20641ec909cd834: Bootstrap starting.
I20260924 04:14:03.299613 5488 tablet_bootstrap.cc:492] T e302fa5313724aad96946d3d105f1690 P 9d2ded6eb21b46f1990eddf60696d213: Bootstrap starting.
I20260924 04:14:03.299863 5489 tablet_bootstrap.cc:492] T e302fa5313724aad96946d3d105f1690 P a44efb30f23a4fcf930c4c03d4d9b5ff: Bootstrap starting.
I20260924 04:14:03.301155 5487 tablet_bootstrap.cc:654] T e302fa5313724aad96946d3d105f1690 P 83d680b6d463476bb20641ec909cd834: Neither blocks nor log segments found. Creating new log.
I20260924 04:14:03.301697 5488 tablet_bootstrap.cc:654] T e302fa5313724aad96946d3d105f1690 P 9d2ded6eb21b46f1990eddf60696d213: Neither blocks nor log segments found. Creating new log.
I20260924 04:14:03.301929 5489 tablet_bootstrap.cc:654] T e302fa5313724aad96946d3d105f1690 P a44efb30f23a4fcf930c4c03d4d9b5ff: Neither blocks nor log segments found. Creating new log.
I20260924 04:14:03.302060 5487 log.cc:824] T e302fa5313724aad96946d3d105f1690 P 83d680b6d463476bb20641ec909cd834: Log is configured to *not* fsync() on all Append() calls
I20260924 04:14:03.302400 5488 log.cc:824] T e302fa5313724aad96946d3d105f1690 P 9d2ded6eb21b46f1990eddf60696d213: Log is configured to *not* fsync() on all Append() calls
I20260924 04:14:03.302721 5489 log.cc:824] T e302fa5313724aad96946d3d105f1690 P a44efb30f23a4fcf930c4c03d4d9b5ff: Log is configured to *not* fsync() on all Append() calls
I20260924 04:14:03.304119 5487 tablet_bootstrap.cc:492] T e302fa5313724aad96946d3d105f1690 P 83d680b6d463476bb20641ec909cd834: No bootstrap required, opened a new log
I20260924 04:14:03.304298 5487 ts_tablet_manager.cc:1397] T e302fa5313724aad96946d3d105f1690 P 83d680b6d463476bb20641ec909cd834: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:14:03.304526 5489 tablet_bootstrap.cc:492] T e302fa5313724aad96946d3d105f1690 P a44efb30f23a4fcf930c4c03d4d9b5ff: No bootstrap required, opened a new log
I20260924 04:14:03.304526 5488 tablet_bootstrap.cc:492] T e302fa5313724aad96946d3d105f1690 P 9d2ded6eb21b46f1990eddf60696d213: No bootstrap required, opened a new log
I20260924 04:14:03.304702 5489 ts_tablet_manager.cc:1397] T e302fa5313724aad96946d3d105f1690 P a44efb30f23a4fcf930c4c03d4d9b5ff: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 04:14:03.304746 5488 ts_tablet_manager.cc:1397] T e302fa5313724aad96946d3d105f1690 P 9d2ded6eb21b46f1990eddf60696d213: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 04:14:03.307634 5487 raft_consensus.cc:348] T e302fa5313724aad96946d3d105f1690 P 83d680b6d463476bb20641ec909cd834 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2ded6eb21b46f1990eddf60696d213" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34451 } } peers { permanent_uuid: "83d680b6d463476bb20641ec909cd834" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39035 } } peers { permanent_uuid: "a44efb30f23a4fcf930c4c03d4d9b5ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35753 } }
I20260924 04:14:03.307928 5487 raft_consensus.cc:374] T e302fa5313724aad96946d3d105f1690 P 83d680b6d463476bb20641ec909cd834 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:14:03.307981 5487 raft_consensus.cc:729] T e302fa5313724aad96946d3d105f1690 P 83d680b6d463476bb20641ec909cd834 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83d680b6d463476bb20641ec909cd834, State: Initialized, Role: FOLLOWER
I20260924 04:14:03.308482 5487 consensus_queue.cc:260] T e302fa5313724aad96946d3d105f1690 P 83d680b6d463476bb20641ec909cd834 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2ded6eb21b46f1990eddf60696d213" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34451 } } peers { permanent_uuid: "83d680b6d463476bb20641ec909cd834" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39035 } } peers { permanent_uuid: "a44efb30f23a4fcf930c4c03d4d9b5ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35753 } }
I20260924 04:14:03.309105 5470 heartbeater.cc:503] Master 127.0.98.62:37891 was elected leader, sending a full tablet report...
I20260924 04:14:03.309028 5489 raft_consensus.cc:348] T e302fa5313724aad96946d3d105f1690 P a44efb30f23a4fcf930c4c03d4d9b5ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2ded6eb21b46f1990eddf60696d213" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34451 } } peers { permanent_uuid: "83d680b6d463476bb20641ec909cd834" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39035 } } peers { permanent_uuid: "a44efb30f23a4fcf930c4c03d4d9b5ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35753 } }
I20260924 04:14:03.309028 5488 raft_consensus.cc:348] T e302fa5313724aad96946d3d105f1690 P 9d2ded6eb21b46f1990eddf60696d213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2ded6eb21b46f1990eddf60696d213" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34451 } } peers { permanent_uuid: "83d680b6d463476bb20641ec909cd834" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39035 } } peers { permanent_uuid: "a44efb30f23a4fcf930c4c03d4d9b5ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35753 } }
I20260924 04:14:03.309263 5489 raft_consensus.cc:374] T e302fa5313724aad96946d3d105f1690 P a44efb30f23a4fcf930c4c03d4d9b5ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:14:03.309263 5488 raft_consensus.cc:374] T e302fa5313724aad96946d3d105f1690 P 9d2ded6eb21b46f1990eddf60696d213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:14:03.309309 5489 raft_consensus.cc:729] T e302fa5313724aad96946d3d105f1690 P a44efb30f23a4fcf930c4c03d4d9b5ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44efb30f23a4fcf930c4c03d4d9b5ff, State: Initialized, Role: FOLLOWER
I20260924 04:14:03.309309 5488 raft_consensus.cc:729] T e302fa5313724aad96946d3d105f1690 P 9d2ded6eb21b46f1990eddf60696d213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d2ded6eb21b46f1990eddf60696d213, State: Initialized, Role: FOLLOWER
I20260924 04:14:03.309478 5487 ts_tablet_manager.cc:1428] T e302fa5313724aad96946d3d105f1690 P 83d680b6d463476bb20641ec909cd834: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 04:14:03.309764 5489 consensus_queue.cc:260] T e302fa5313724aad96946d3d105f1690 P a44efb30f23a4fcf930c4c03d4d9b5ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2ded6eb21b46f1990eddf60696d213" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34451 } } peers { permanent_uuid: "83d680b6d463476bb20641ec909cd834" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39035 } } peers { permanent_uuid: "a44efb30f23a4fcf930c4c03d4d9b5ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35753 } }
I20260924 04:14:03.309764 5488 consensus_queue.cc:260] T e302fa5313724aad96946d3d105f1690 P 9d2ded6eb21b46f1990eddf60696d213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2ded6eb21b46f1990eddf60696d213" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34451 } } peers { permanent_uuid: "83d680b6d463476bb20641ec909cd834" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39035 } } peers { permanent_uuid: "a44efb30f23a4fcf930c4c03d4d9b5ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35753 } }
I20260924 04:14:03.310698 5489 ts_tablet_manager.cc:1428] T e302fa5313724aad96946d3d105f1690 P a44efb30f23a4fcf930c4c03d4d9b5ff: Time spent starting tablet: real 0.006s user 0.000s sys 0.007s
I20260924 04:14:03.310714 5208 heartbeater.cc:503] Master 127.0.98.62:37891 was elected leader, sending a full tablet report...
I20260924 04:14:03.310920 5339 heartbeater.cc:503] Master 127.0.98.62:37891 was elected leader, sending a full tablet report...
I20260924 04:14:03.310703 5488 ts_tablet_manager.cc:1428] T e302fa5313724aad96946d3d105f1690 P 9d2ded6eb21b46f1990eddf60696d213: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260924 04:14:03.331984 5340 tablet.cc:2369] T e302fa5313724aad96946d3d105f1690 P a44efb30f23a4fcf930c4c03d4d9b5ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:14:03.414013 5209 tablet.cc:2369] T e302fa5313724aad96946d3d105f1690 P 9d2ded6eb21b46f1990eddf60696d213: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:14:03.445358 5494 raft_consensus.cc:482] T e302fa5313724aad96946d3d105f1690 P a44efb30f23a4fcf930c4c03d4d9b5ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:14:03.445606 5494 raft_consensus.cc:504] T e302fa5313724aad96946d3d105f1690 P a44efb30f23a4fcf930c4c03d4d9b5ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2ded6eb21b46f1990eddf60696d213" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34451 } } peers { permanent_uuid: "83d680b6d463476bb20641ec909cd834" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39035 } } peers { permanent_uuid: "a44efb30f23a4fcf930c4c03d4d9b5ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35753 } }
I20260924 04:14:03.446811 5494 leader_election.cc:288] T e302fa5313724aad96946d3d105f1690 P a44efb30f23a4fcf930c4c03d4d9b5ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d2ded6eb21b46f1990eddf60696d213 (127.0.98.1:34451), 83d680b6d463476bb20641ec909cd834 (127.0.98.3:39035)
I20260924 04:14:03.448969 5341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:03.447068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34451 (local address 127.0.98.2:54601)
0924 04:14:03.447289 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:03.447303 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:03.447423 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:03.447693 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:03.447700 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:03.447716 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:03.448290 (+ 574us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:14:03.448291 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:03.448302 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:14:03.448326 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:14:03.448709 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:14:03.448712 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:14:03.448715 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 04:14:03.448806 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 04:14:03.448844 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123}
I20260924 04:14:03.449007 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:03.447160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54601 (local address 127.0.98.1:34451)
0924 04:14:03.447311 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:03.447316 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:03.447383 (+ 67us) server_negotiation.cc:407] Connection header received
0924 04:14:03.447528 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:03.447531 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:03.447578 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:03.447677 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:03.448520 (+ 843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:03.448524 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:03.448526 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:14:03.448540 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:03.448566 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:14:03.448576 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:03.448653 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:03.448809 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 04:14:03.448881 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:14:03.449044 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:03.447165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51243 (local address 127.0.98.3:39035)
0924 04:14:03.447311 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:03.447316 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:03.447649 (+ 333us) server_negotiation.cc:407] Connection header received
0924 04:14:03.447758 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:03.447761 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:03.447801 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:03.447885 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:03.448663 (+ 778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:03.448667 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:03.448670 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:14:03.448684 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:03.448705 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:14:03.448715 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:03.448809 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:03.448900 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 04:14:03.448937 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 04:14:03.449095 5496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:03.447068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39035 (local address 127.0.98.2:51243)
0924 04:14:03.447527 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:03.447542 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:03.447665 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:03.447947 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:03.447950 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:03.447965 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:03.448377 (+ 412us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:14:03.448378 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:03.448390 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:14:03.448483 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:14:03.448807 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:14:03.448810 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:14:03.448812 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 04:14:03.448889 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 04:14:03.448921 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"mutex_wait_us":53,"thread_start_us":150,"threads_started":1}
I20260924 04:14:03.449688 5424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e302fa5313724aad96946d3d105f1690" candidate_uuid: "a44efb30f23a4fcf930c4c03d4d9b5ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83d680b6d463476bb20641ec909cd834" is_pre_election: true
I20260924 04:14:03.449731 5162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e302fa5313724aad96946d3d105f1690" candidate_uuid: "a44efb30f23a4fcf930c4c03d4d9b5ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d2ded6eb21b46f1990eddf60696d213" is_pre_election: true
I20260924 04:14:03.449963 5424 raft_consensus.cc:2454] T e302fa5313724aad96946d3d105f1690 P 83d680b6d463476bb20641ec909cd834 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44efb30f23a4fcf930c4c03d4d9b5ff in term 0.
I20260924 04:14:03.449997 5162 raft_consensus.cc:2454] T e302fa5313724aad96946d3d105f1690 P 9d2ded6eb21b46f1990eddf60696d213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44efb30f23a4fcf930c4c03d4d9b5ff in term 0.
I20260924 04:14:03.450423 5228 leader_election.cc:302] T e302fa5313724aad96946d3d105f1690 P a44efb30f23a4fcf930c4c03d4d9b5ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d2ded6eb21b46f1990eddf60696d213, a44efb30f23a4fcf930c4c03d4d9b5ff; no voters:
I20260924 04:14:03.450688 5494 raft_consensus.cc:2781] T e302fa5313724aad96946d3d105f1690 P a44efb30f23a4fcf930c4c03d4d9b5ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:14:03.450783 5494 raft_consensus.cc:482] T e302fa5313724aad96946d3d105f1690 P a44efb30f23a4fcf930c4c03d4d9b5ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:14:03.450817 5494 raft_consensus.cc:3037] T e302fa5313724aad96946d3d105f1690 P a44efb30f23a4fcf930c4c03d4d9b5ff [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:14:03.451889 5494 raft_consensus.cc:504] T e302fa5313724aad96946d3d105f1690 P a44efb30f23a4fcf930c4c03d4d9b5ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2ded6eb21b46f1990eddf60696d213" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34451 } } peers { permanent_uuid: "83d680b6d463476bb20641ec909cd834" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39035 } } peers { permanent_uuid: "a44efb30f23a4fcf930c4c03d4d9b5ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35753 } }
I20260924 04:14:03.452224 5494 leader_election.cc:288] T e302fa5313724aad96946d3d105f1690 P a44efb30f23a4fcf930c4c03d4d9b5ff [CANDIDATE]: Term 1 election: Requested vote from peers 9d2ded6eb21b46f1990eddf60696d213 (127.0.98.1:34451), 83d680b6d463476bb20641ec909cd834 (127.0.98.3:39035)
I20260924 04:14:03.452602 5424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e302fa5313724aad96946d3d105f1690" candidate_uuid: "a44efb30f23a4fcf930c4c03d4d9b5ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83d680b6d463476bb20641ec909cd834"
I20260924 04:14:03.452607 5162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e302fa5313724aad96946d3d105f1690" candidate_uuid: "a44efb30f23a4fcf930c4c03d4d9b5ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d2ded6eb21b46f1990eddf60696d213"
I20260924 04:14:03.452734 5424 raft_consensus.cc:3037] T e302fa5313724aad96946d3d105f1690 P 83d680b6d463476bb20641ec909cd834 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:14:03.452775 5162 raft_consensus.cc:3037] T e302fa5313724aad96946d3d105f1690 P 9d2ded6eb21b46f1990eddf60696d213 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:14:03.453986 5162 raft_consensus.cc:2454] T e302fa5313724aad96946d3d105f1690 P 9d2ded6eb21b46f1990eddf60696d213 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44efb30f23a4fcf930c4c03d4d9b5ff in term 1.
I20260924 04:14:03.453986 5424 raft_consensus.cc:2454] T e302fa5313724aad96946d3d105f1690 P 83d680b6d463476bb20641ec909cd834 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44efb30f23a4fcf930c4c03d4d9b5ff in term 1.
I20260924 04:14:03.454278 5228 leader_election.cc:302] T e302fa5313724aad96946d3d105f1690 P a44efb30f23a4fcf930c4c03d4d9b5ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d2ded6eb21b46f1990eddf60696d213, a44efb30f23a4fcf930c4c03d4d9b5ff; no voters:
I20260924 04:14:03.454438 5494 raft_consensus.cc:2781] T e302fa5313724aad96946d3d105f1690 P a44efb30f23a4fcf930c4c03d4d9b5ff [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:14:03.454653 5494 raft_consensus.cc:686] T e302fa5313724aad96946d3d105f1690 P a44efb30f23a4fcf930c4c03d4d9b5ff [term 1 LEADER]: Becoming Leader. State: Replica: a44efb30f23a4fcf930c4c03d4d9b5ff, State: Running, Role: LEADER
I20260924 04:14:03.455029 5494 consensus_queue.cc:237] T e302fa5313724aad96946d3d105f1690 P a44efb30f23a4fcf930c4c03d4d9b5ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2ded6eb21b46f1990eddf60696d213" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34451 } } peers { permanent_uuid: "83d680b6d463476bb20641ec909cd834" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39035 } } peers { permanent_uuid: "a44efb30f23a4fcf930c4c03d4d9b5ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35753 } }
I20260924 04:14:03.458158 5019 catalog_manager.cc:5495] T e302fa5313724aad96946d3d105f1690 P a44efb30f23a4fcf930c4c03d4d9b5ff reported cstate change: term changed from 0 to 1, leader changed from <none> to a44efb30f23a4fcf930c4c03d4d9b5ff (127.0.98.2). New cstate: current_term: 1 leader_uuid: "a44efb30f23a4fcf930c4c03d4d9b5ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2ded6eb21b46f1990eddf60696d213" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83d680b6d463476bb20641ec909cd834" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a44efb30f23a4fcf930c4c03d4d9b5ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35753 } health_report { overall_health: HEALTHY } } }
I20260924 04:14:03.478582 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:03.477242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54410 (local address 127.0.98.2:35753)
0924 04:14:03.477366 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:03.477369 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:03.477383 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:14:03.477492 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:03.477494 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:03.477553 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:03.477622 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:03.478262 (+ 640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:03.478265 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:03.478267 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:14:03.478278 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:03.478297 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:14:03.478305 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:03.478375 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:03.478422 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 04:14:03.478453 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 04:14:03.484375 5424 raft_consensus.cc:1261] T e302fa5313724aad96946d3d105f1690 P 83d680b6d463476bb20641ec909cd834 [term 1 FOLLOWER]: Refusing update from remote peer a44efb30f23a4fcf930c4c03d4d9b5ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:14:03.484375 5162 raft_consensus.cc:1261] T e302fa5313724aad96946d3d105f1690 P 9d2ded6eb21b46f1990eddf60696d213 [term 1 FOLLOWER]: Refusing update from remote peer a44efb30f23a4fcf930c4c03d4d9b5ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:14:03.485029 5494 consensus_queue.cc:1035] T e302fa5313724aad96946d3d105f1690 P a44efb30f23a4fcf930c4c03d4d9b5ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "83d680b6d463476bb20641ec909cd834" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:14:03.485208 5497 consensus_queue.cc:1035] T e302fa5313724aad96946d3d105f1690 P a44efb30f23a4fcf930c4c03d4d9b5ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d2ded6eb21b46f1990eddf60696d213" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:14:03.492515 5501 mvcc.cc:205] Tried to move back new op lower bound from 7332754405304958976 to 7332754405194391552. Current Snapshot: MvccSnapshot[applied={T|T < 7332754405304958976}]
I20260924 04:14:03.492980 5500 mvcc.cc:205] Tried to move back new op lower bound from 7332754405304958976 to 7332754405194391552. Current Snapshot: MvccSnapshot[applied={T|T < 7332754405304958976}]
I20260924 04:14:03.494482 5502 mvcc.cc:205] Tried to move back new op lower bound from 7332754405304958976 to 7332754405194391552. Current Snapshot: MvccSnapshot[applied={T|T < 7332754405304958976}]
I20260924 04:14:03.500200 5018 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:14:03.500371 5018 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:14:03.502640 5018 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 93ff8b95282f4fdd8f6214e036efaf4e: Sending DeleteTablet for 3 replicas of tablet e302fa5313724aad96946d3d105f1690
I20260924 04:14:03.503381 5142 tablet_service.cc:1514] Processing DeleteTablet for tablet e302fa5313724aad96946d3d105f1690 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:14:03 UTC) from {username='slave'} at 127.0.0.1:36250
I20260924 04:14:03.503576 5404 tablet_service.cc:1514] Processing DeleteTablet for tablet e302fa5313724aad96946d3d105f1690 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:14:03 UTC) from {username='slave'} at 127.0.0.1:47832
I20260924 04:14:03.503896 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 5080
I20260924 04:14:03.503973 5273 tablet_service.cc:1514] Processing DeleteTablet for tablet e302fa5313724aad96946d3d105f1690 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:14:03 UTC) from {username='slave'} at 127.0.0.1:54408
I20260924 04:14:03.504326 5511 tablet_replica.cc:330] T e302fa5313724aad96946d3d105f1690 P 83d680b6d463476bb20641ec909cd834: stopping tablet replica
I20260924 04:14:03.504568 5511 raft_consensus.cc:2229] T e302fa5313724aad96946d3d105f1690 P 83d680b6d463476bb20641ec909cd834 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:14:03.504825 5511 raft_consensus.cc:2258] T e302fa5313724aad96946d3d105f1690 P 83d680b6d463476bb20641ec909cd834 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:14:03.505081 5512 tablet_replica.cc:330] T e302fa5313724aad96946d3d105f1690 P a44efb30f23a4fcf930c4c03d4d9b5ff: stopping tablet replica
I20260924 04:14:03.505317 5512 raft_consensus.cc:2229] T e302fa5313724aad96946d3d105f1690 P a44efb30f23a4fcf930c4c03d4d9b5ff [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:14:03.505663 5512 raft_consensus.cc:2258] T e302fa5313724aad96946d3d105f1690 P a44efb30f23a4fcf930c4c03d4d9b5ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:14:03.506326 5511 ts_tablet_manager.cc:1905] T e302fa5313724aad96946d3d105f1690 P 83d680b6d463476bb20641ec909cd834: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:14:03.506858 5512 ts_tablet_manager.cc:1905] T e302fa5313724aad96946d3d105f1690 P a44efb30f23a4fcf930c4c03d4d9b5ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:14:03.508952 5511 ts_tablet_manager.cc:1918] T e302fa5313724aad96946d3d105f1690 P 83d680b6d463476bb20641ec909cd834: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:14:03.509078 5511 log.cc:1196] T e302fa5313724aad96946d3d105f1690 P 83d680b6d463476bb20641ec909cd834: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/ts-2/wal/wals/e302fa5313724aad96946d3d105f1690
I20260924 04:14:03.509261 5512 ts_tablet_manager.cc:1918] T e302fa5313724aad96946d3d105f1690 P a44efb30f23a4fcf930c4c03d4d9b5ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:14:03.509375 5512 log.cc:1196] T e302fa5313724aad96946d3d105f1690 P a44efb30f23a4fcf930c4c03d4d9b5ff: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790223216265354-392-0/minicluster-data/ts-1/wal/wals/e302fa5313724aad96946d3d105f1690
I20260924 04:14:03.509400 5511 ts_tablet_manager.cc:1939] T e302fa5313724aad96946d3d105f1690 P 83d680b6d463476bb20641ec909cd834: Deleting consensus metadata
I20260924 04:14:03.509677 5512 ts_tablet_manager.cc:1939] T e302fa5313724aad96946d3d105f1690 P a44efb30f23a4fcf930c4c03d4d9b5ff: Deleting consensus metadata
I20260924 04:14:03.510641 5007 catalog_manager.cc:4841] TS a44efb30f23a4fcf930c4c03d4d9b5ff (127.0.98.2:35753): tablet e302fa5313724aad96946d3d105f1690 (table test-table [id=69f5532a111e45978f8c42ee78bf79b2]) successfully deleted
I20260924 04:14:03.510820 5007 catalog_manager.cc:4841] TS 83d680b6d463476bb20641ec909cd834 (127.0.98.3:39035): tablet e302fa5313724aad96946d3d105f1690 (table test-table [id=69f5532a111e45978f8c42ee78bf79b2]) successfully deleted
W20260924 04:14:03.513069 5006 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34451 (error 108)
I20260924 04:14:03.513286 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 5211
I20260924 04:14:03.515100 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:03.514803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34451 (local address 127.0.0.1:36254)
0924 04:14:03.514942 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:03.515014 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 04:14:03.515276 5006 catalog_manager.cc:4584] TS 9d2ded6eb21b46f1990eddf60696d213 (127.0.98.1:34451): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e302fa5313724aad96946d3d105f1690: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34451: connect: Connection refused (error 111)
I20260924 04:14:03.518390 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 5342
I20260924 04:14:03.522820 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 4989
2026-09-24T04:14:03Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1038 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260924 04:14:03.542104 392 mini_oidc.cc:152] Starting JWKS server
I20260924 04:14:03.543648 392 webserver.cc:460] Webserver started at https://127.0.0.1:44449/ using document root <none> and password file <none>
I20260924 04:14:03.543785 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 04:14:03.544030 392 webserver.cc:460] Webserver started at http://127.0.0.1:43233/ using document root <none> and password file <none>
2026-09-24T04:14:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:14:03Z Disabled control of system clock
I20260924 04:14:03.552130 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790223216265354-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790223216265354-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790223216265354-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33723
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33723
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44449/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790223216265354-392-0/testchainca.cert with env {}
W20260924 04:14:03.661794 5519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:14:03.662084 5519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:14:03.662127 5519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:14:03.666169 5519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:14:03.666241 5519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:14:03.666258 5519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:14:03.666276 5519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:14:03.666293 5519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 04:14:03.666318 5519 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 04:14:03.666333 5519 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44449/jwks.json
I20260924 04:14:03.671873 5519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790223216265354-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790223216265354-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33723
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44449/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790223216265354-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790223216265354-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:14:03.673098 5519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:14:03.674145 5519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:14:03.682557 5524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:03.682705 5525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:03.682756 5527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:03.683046 5519 server_base.cc:1034] running on GCE node
I20260924 04:14:03.683739 5519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:14:03.684914 5519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:14:03.686100 5519 hybrid_clock.cc:648] HybridClock initialized: now 1790223243686079 us; error 30 us; skew 500 ppm
I20260924 04:14:03.687940 5519 webserver.cc:460] Webserver started at http://127.0.98.62:40291/ using document root <none> and password file <none>
I20260924 04:14:03.688469 5519 fs_manager.cc:359] Metadata directory not provided
I20260924 04:14:03.688526 5519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:14:03.688681 5519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:14:03.690649 5519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790223216265354-392-0/minicluster-data/master-0/data/instance:
uuid: "5fb563a5cc4b4cfdba997c967199a69b"
format_stamp: "Formatted at 2026-09-24 04:14:03 on dist-test-slave-p401"
server_key: "f2d0b73e635d581d1dc376a25fabc59f"
server_key_iv: "2719e8165139e172cf786e553261b62d"
server_key_version: "encryptionkey@0"
I20260924 04:14:03.691154 5519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790223216265354-392-0/minicluster-data/master-0/wal/instance:
uuid: "5fb563a5cc4b4cfdba997c967199a69b"
format_stamp: "Formatted at 2026-09-24 04:14:03 on dist-test-slave-p401"
server_key: "f2d0b73e635d581d1dc376a25fabc59f"
server_key_iv: "2719e8165139e172cf786e553261b62d"
server_key_version: "encryptionkey@0"
I20260924 04:14:03.694770 5519 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:14:03.697145 5533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:03.698446 5519 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:14:03.698553 5519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790223216265354-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790223216265354-392-0/minicluster-data/master-0/wal
uuid: "5fb563a5cc4b4cfdba997c967199a69b"
format_stamp: "Formatted at 2026-09-24 04:14:03 on dist-test-slave-p401"
server_key: "f2d0b73e635d581d1dc376a25fabc59f"
server_key_iv: "2719e8165139e172cf786e553261b62d"
server_key_version: "encryptionkey@0"
I20260924 04:14:03.698669 5519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790223216265354-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790223216265354-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790223216265354-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:14:03.718811 5519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:14:03.719568 5519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:14:03.719740 5519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:14:03.726380 5519 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33723
I20260924 04:14:03.726402 5585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33723 every 8 connection(s)
I20260924 04:14:03.727330 5519 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:14:03.730412 5586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:14:03.735723 5586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fb563a5cc4b4cfdba997c967199a69b: Bootstrap starting.
I20260924 04:14:03.737390 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 5519
I20260924 04:14:03.737526 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790223216265354-392-0/minicluster-data/master-0/wal/instance
I20260924 04:14:03.737798 392 external_mini_cluster.cc:1428] Setting key d8fa9d144977723737e95c887581efb5
I20260924 04:14:03.738267 5586 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5fb563a5cc4b4cfdba997c967199a69b: Neither blocks nor log segments found. Creating new log.
I20260924 04:14:03.739069 5586 log.cc:824] T 00000000000000000000000000000000 P 5fb563a5cc4b4cfdba997c967199a69b: Log is configured to *not* fsync() on all Append() calls
I20260924 04:14:03.740875 5586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fb563a5cc4b4cfdba997c967199a69b: No bootstrap required, opened a new log
I20260924 04:14:03.744278 5586 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5fb563a5cc4b4cfdba997c967199a69b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fb563a5cc4b4cfdba997c967199a69b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33723 } }
I20260924 04:14:03.744503 5586 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5fb563a5cc4b4cfdba997c967199a69b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:14:03.744558 5586 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5fb563a5cc4b4cfdba997c967199a69b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fb563a5cc4b4cfdba997c967199a69b, State: Initialized, Role: FOLLOWER
I20260924 04:14:03.744967 5586 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fb563a5cc4b4cfdba997c967199a69b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fb563a5cc4b4cfdba997c967199a69b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33723 } }
I20260924 04:14:03.745098 5586 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5fb563a5cc4b4cfdba997c967199a69b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:14:03.745159 5586 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5fb563a5cc4b4cfdba997c967199a69b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:14:03.745258 5586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5fb563a5cc4b4cfdba997c967199a69b [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:14:03.746071 5586 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5fb563a5cc4b4cfdba997c967199a69b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fb563a5cc4b4cfdba997c967199a69b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33723 } }
I20260924 04:14:03.746381 5586 leader_election.cc:302] T 00000000000000000000000000000000 P 5fb563a5cc4b4cfdba997c967199a69b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fb563a5cc4b4cfdba997c967199a69b; no voters:
I20260924 04:14:03.746424 5589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:03.738853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51002 (local address 127.0.98.62:33723)
0924 04:14:03.739509 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:03.739524 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:03.739559 (+ 35us) server_negotiation.cc:407] Connection header received
0924 04:14:03.740166 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:03.740184 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:03.740519 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:03.740834 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:03.741240 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:03.742187 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:03.743048 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:03.743533 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:03.743672 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:03.744514 (+ 842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:03.744520 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:03.744536 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:14:03.744991 (+ 455us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:03.745044 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:14:03.745064 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:03.745445 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:03.745501 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 04:14:03.745706 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":117,"threads_started":1}
I20260924 04:14:03.746659 5586 leader_election.cc:288] T 00000000000000000000000000000000 P 5fb563a5cc4b4cfdba997c967199a69b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:14:03.746846 5591 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5fb563a5cc4b4cfdba997c967199a69b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:14:03.747020 5591 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5fb563a5cc4b4cfdba997c967199a69b [term 1 LEADER]: Becoming Leader. State: Replica: 5fb563a5cc4b4cfdba997c967199a69b, State: Running, Role: LEADER
I20260924 04:14:03.747366 5591 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5fb563a5cc4b4cfdba997c967199a69b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fb563a5cc4b4cfdba997c967199a69b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33723 } }
I20260924 04:14:03.747850 5586 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5fb563a5cc4b4cfdba997c967199a69b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:14:03.748970 5593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fb563a5cc4b4cfdba997c967199a69b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5fb563a5cc4b4cfdba997c967199a69b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fb563a5cc4b4cfdba997c967199a69b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33723 } } }
I20260924 04:14:03.749087 5593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fb563a5cc4b4cfdba997c967199a69b [sys.catalog]: This master's current role is: LEADER
I20260924 04:14:03.749461 5600 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:14:03.749948 5592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fb563a5cc4b4cfdba997c967199a69b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fb563a5cc4b4cfdba997c967199a69b. Latest consensus state: current_term: 1 leader_uuid: "5fb563a5cc4b4cfdba997c967199a69b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fb563a5cc4b4cfdba997c967199a69b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33723 } } }
I20260924 04:14:03.750161 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:14:03.750160 5592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fb563a5cc4b4cfdba997c967199a69b [sys.catalog]: This master's current role is: LEADER
I20260924 04:14:03.752424 5600 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:14:03.756979 5600 catalog_manager.cc:1347] Generated new cluster ID: b3cd7d8ca1f9430ea13472d5fef8a782
I20260924 04:14:03.757054 5600 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:14:03.772702 5600 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:14:03.773599 5600 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:14:03.781627 5600 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5fb563a5cc4b4cfdba997c967199a69b: Generated new TSK 0
I20260924 04:14:03.782461 5600 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 04:14:03.821693 5616 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260924 04:14:03.821815 5616 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 04:14:03.822299 5589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:03.798688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51012 (local address 127.0.98.62:33723)
0924 04:14:03.798808 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:03.798812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:03.799005 (+ 193us) server_negotiation.cc:407] Connection header received
0924 04:14:03.799214 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:03.799218 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:03.799296 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:03.799381 (+ 85us) server_negotiation.cc:226] Negotiated authn=JWT
0924 04:14:03.799622 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:03.800158 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:03.800995 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:03.801211 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:03.801293 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:03.801604 (+ 311us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 04:14:03.821865 (+ 20261us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44449/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 04:14:03.822156 (+ 291us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51012: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44449/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":39}
W20260924 04:14:03.822399 5589 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51012: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44449/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260924 04:14:03.823545 5615 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:03.798633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33723 (local address 127.0.0.1:51012)
0924 04:14:03.798928 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:03.798940 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:03.799060 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:03.799405 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:03.799408 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:03.799416 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0924 04:14:03.799526 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:03.799531 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:03.800263 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:03.800267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:03.800891 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:03.800898 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:03.801293 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:03.801310 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:03.801399 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:03.801439 (+ 40us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 04:14:03.822094 (+ 20655us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44449/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 04:14:03.822360 (+ 266us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:33723: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44449/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":101,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:33723: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44449/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260924 04:14:03.824164 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 5519
2026-09-24T04:14:03Z chronyd exiting
I20260924 04:14:03.860580 392 test_util.cc:175] -----------------------------------------------
I20260924 04:14:03.860671 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790223216265354-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (319 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260924 04:14:03.861689 392 mini_oidc.cc:152] Starting JWKS server
I20260924 04:14:03.862519 392 webserver.cc:460] Webserver started at https://127.0.0.1:45973/ using document root <none> and password file <none>
I20260924 04:14:03.862612 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 04:14:03.862856 392 webserver.cc:460] Webserver started at http://127.0.0.1:42717/ using document root <none> and password file <none>
2026-09-24T04:14:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:14:03Z Disabled control of system clock
I20260924 04:14:03.870870 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790223216265354-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790223216265354-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790223216265354-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38609
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38609
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45973/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790223216265354-392-0/testchainca.cert with env {}
W20260924 04:14:03.976533 5622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:14:03.976831 5622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:14:03.976889 5622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:14:03.980999 5622 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:14:03.981066 5622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:14:03.981089 5622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:14:03.981110 5622 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:14:03.981132 5622 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 04:14:03.981156 5622 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 04:14:03.981184 5622 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45973/jwks.json
I20260924 04:14:03.986713 5622 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790223216265354-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790223216265354-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38609
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45973/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790223216265354-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790223216265354-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:14:03.987816 5622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:14:03.988838 5622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:14:03.996939 5627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:03.997059 5630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:03.996992 5628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:03.997133 5622 server_base.cc:1034] running on GCE node
I20260924 04:14:03.997728 5622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:14:03.998793 5622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:14:03.999953 5622 hybrid_clock.cc:648] HybridClock initialized: now 1790223243999931 us; error 35 us; skew 500 ppm
I20260924 04:14:04.001693 5622 webserver.cc:460] Webserver started at http://127.0.98.62:45851/ using document root <none> and password file <none>
I20260924 04:14:04.002224 5622 fs_manager.cc:359] Metadata directory not provided
I20260924 04:14:04.002288 5622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:14:04.002475 5622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:14:04.004453 5622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790223216265354-392-0/minicluster-data/master-0/data/instance:
uuid: "534b3fa22489418ea812e60511ccca2f"
format_stamp: "Formatted at 2026-09-24 04:14:03 on dist-test-slave-p401"
server_key: "1b226ef0047754d79d31393f0feb5730"
server_key_iv: "3f8a1b313dd1706cf1d482feb2b7a742"
server_key_version: "encryptionkey@0"
I20260924 04:14:04.004985 5622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790223216265354-392-0/minicluster-data/master-0/wal/instance:
uuid: "534b3fa22489418ea812e60511ccca2f"
format_stamp: "Formatted at 2026-09-24 04:14:03 on dist-test-slave-p401"
server_key: "1b226ef0047754d79d31393f0feb5730"
server_key_iv: "3f8a1b313dd1706cf1d482feb2b7a742"
server_key_version: "encryptionkey@0"
I20260924 04:14:04.008569 5622 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:14:04.011107 5636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:04.012507 5622 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:14:04.012633 5622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790223216265354-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790223216265354-392-0/minicluster-data/master-0/wal
uuid: "534b3fa22489418ea812e60511ccca2f"
format_stamp: "Formatted at 2026-09-24 04:14:03 on dist-test-slave-p401"
server_key: "1b226ef0047754d79d31393f0feb5730"
server_key_iv: "3f8a1b313dd1706cf1d482feb2b7a742"
server_key_version: "encryptionkey@0"
I20260924 04:14:04.012744 5622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790223216265354-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790223216265354-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790223216265354-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:14:04.031910 5622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:14:04.032605 5622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:14:04.032774 5622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:14:04.039340 5622 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38609
I20260924 04:14:04.039327 5688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38609 every 8 connection(s)
I20260924 04:14:04.040330 5622 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:14:04.043421 5689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:14:04.046048 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 5622
I20260924 04:14:04.046171 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790223216265354-392-0/minicluster-data/master-0/wal/instance
I20260924 04:14:04.046456 392 external_mini_cluster.cc:1428] Setting key 310844da2e5d7efdb71b131525c17d1a
I20260924 04:14:04.049269 5689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 534b3fa22489418ea812e60511ccca2f: Bootstrap starting.
I20260924 04:14:04.051674 5689 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 534b3fa22489418ea812e60511ccca2f: Neither blocks nor log segments found. Creating new log.
I20260924 04:14:04.052562 5689 log.cc:824] T 00000000000000000000000000000000 P 534b3fa22489418ea812e60511ccca2f: Log is configured to *not* fsync() on all Append() calls
I20260924 04:14:04.054350 5689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 534b3fa22489418ea812e60511ccca2f: No bootstrap required, opened a new log
I20260924 04:14:04.055639 5692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:04.047857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40416 (local address 127.0.98.62:38609)
0924 04:14:04.048475 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:04.048490 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:04.048525 (+ 35us) server_negotiation.cc:407] Connection header received
0924 04:14:04.049137 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:04.049163 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:04.049587 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:04.049880 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:04.050235 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:04.051206 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:04.052416 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:04.052977 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:04.053129 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:04.053891 (+ 762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:04.053895 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:04.053907 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:14:04.054298 (+ 391us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:04.054350 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:14:04.054369 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:04.054461 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:04.054614 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 04:14:04.054886 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":104,"threads_started":1}
I20260924 04:14:04.057987 5689 raft_consensus.cc:348] T 00000000000000000000000000000000 P 534b3fa22489418ea812e60511ccca2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "534b3fa22489418ea812e60511ccca2f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38609 } }
I20260924 04:14:04.058210 5689 raft_consensus.cc:374] T 00000000000000000000000000000000 P 534b3fa22489418ea812e60511ccca2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:14:04.058234 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:14:04.058269 5689 raft_consensus.cc:729] T 00000000000000000000000000000000 P 534b3fa22489418ea812e60511ccca2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 534b3fa22489418ea812e60511ccca2f, State: Initialized, Role: FOLLOWER
I20260924 04:14:04.058719 5689 consensus_queue.cc:260] T 00000000000000000000000000000000 P 534b3fa22489418ea812e60511ccca2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "534b3fa22489418ea812e60511ccca2f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38609 } }
I20260924 04:14:04.058869 5689 raft_consensus.cc:388] T 00000000000000000000000000000000 P 534b3fa22489418ea812e60511ccca2f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:14:04.058933 5689 raft_consensus.cc:482] T 00000000000000000000000000000000 P 534b3fa22489418ea812e60511ccca2f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:14:04.059026 5689 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 534b3fa22489418ea812e60511ccca2f [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:14:04.059927 5689 raft_consensus.cc:504] T 00000000000000000000000000000000 P 534b3fa22489418ea812e60511ccca2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "534b3fa22489418ea812e60511ccca2f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38609 } }
I20260924 04:14:04.060248 5689 leader_election.cc:302] T 00000000000000000000000000000000 P 534b3fa22489418ea812e60511ccca2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 534b3fa22489418ea812e60511ccca2f; no voters:
I20260924 04:14:04.060519 5689 leader_election.cc:288] T 00000000000000000000000000000000 P 534b3fa22489418ea812e60511ccca2f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:14:04.060734 5694 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 534b3fa22489418ea812e60511ccca2f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:14:04.060948 5694 raft_consensus.cc:686] T 00000000000000000000000000000000 P 534b3fa22489418ea812e60511ccca2f [term 1 LEADER]: Becoming Leader. State: Replica: 534b3fa22489418ea812e60511ccca2f, State: Running, Role: LEADER
I20260924 04:14:04.061264 5694 consensus_queue.cc:237] T 00000000000000000000000000000000 P 534b3fa22489418ea812e60511ccca2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "534b3fa22489418ea812e60511ccca2f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38609 } }
I20260924 04:14:04.061625 5689 sys_catalog.cc:564] T 00000000000000000000000000000000 P 534b3fa22489418ea812e60511ccca2f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:14:04.062757 5696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 534b3fa22489418ea812e60511ccca2f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 534b3fa22489418ea812e60511ccca2f. Latest consensus state: current_term: 1 leader_uuid: "534b3fa22489418ea812e60511ccca2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "534b3fa22489418ea812e60511ccca2f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38609 } } }
I20260924 04:14:04.062877 5696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 534b3fa22489418ea812e60511ccca2f [sys.catalog]: This master's current role is: LEADER
I20260924 04:14:04.063290 5703 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:14:04.063277 5695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 534b3fa22489418ea812e60511ccca2f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "534b3fa22489418ea812e60511ccca2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "534b3fa22489418ea812e60511ccca2f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38609 } } }
I20260924 04:14:04.063434 5695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 534b3fa22489418ea812e60511ccca2f [sys.catalog]: This master's current role is: LEADER
I20260924 04:14:04.066428 5703 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:14:04.072574 5703 catalog_manager.cc:1347] Generated new cluster ID: 7f9e743d8507459bbadadd1e0637701d
I20260924 04:14:04.072710 5703 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:14:04.078871 5703 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:14:04.079813 5703 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:14:04.087775 5703 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 534b3fa22489418ea812e60511ccca2f: Generated new TSK 0
I20260924 04:14:04.088593 5703 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 04:14:04.123883 5719 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260924 04:14:04.124064 5719 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 04:14:04.124665 5692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:04.102779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40428 (local address 127.0.98.62:38609)
0924 04:14:04.102904 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:04.102908 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:04.103132 (+ 224us) server_negotiation.cc:407] Connection header received
0924 04:14:04.103343 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:04.103347 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:04.103413 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:04.103536 (+ 123us) server_negotiation.cc:226] Negotiated authn=JWT
0924 04:14:04.103787 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:04.104580 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:04.105861 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:04.106195 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:04.106267 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:04.106753 (+ 486us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 04:14:04.124312 (+ 17559us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45973/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 04:14:04.124553 (+ 241us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40428: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45973/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":49}
W20260924 04:14:04.124723 5692 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40428: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45973/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260924 04:14:04.124920 5718 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:04.102725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38609 (local address 127.0.0.1:40428)
0924 04:14:04.103045 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:04.103060 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:04.103178 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:04.103542 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:04.103545 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:04.103554 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0924 04:14:04.103673 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:04.103679 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:04.104739 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:04.104742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:04.105716 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:04.105727 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:04.106389 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:04.106412 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:04.106495 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:04.106525 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 04:14:04.124535 (+ 18010us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45973/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 04:14:04.124739 (+ 204us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:38609: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45973/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":114,"threads_started":1}
I20260924 04:14:04.125499 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 5622
2026-09-24T04:14:04Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (304 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260924 04:14:04.165932 392 mini_oidc.cc:152] Starting JWKS server
I20260924 04:14:04.166788 392 webserver.cc:460] Webserver started at https://127.0.0.1:43861/ using document root <none> and password file <none>
I20260924 04:14:04.166886 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 04:14:04.167109 392 webserver.cc:460] Webserver started at http://127.0.0.1:45143/ using document root <none> and password file <none>
2026-09-24T04:14:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:14:04Z Disabled control of system clock
I20260924 04:14:04.175184 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790223216265354-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790223216265354-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790223216265354-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45915
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45915
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43861/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 04:14:04.283771 5725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:14:04.284054 5725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:14:04.284111 5725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:14:04.288220 5725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:14:04.288292 5725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:14:04.288313 5725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:14:04.288335 5725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:14:04.288364 5725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 04:14:04.288395 5725 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 04:14:04.288421 5725 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43861/jwks.json
I20260924 04:14:04.293439 5725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790223216265354-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790223216265354-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45915
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43861/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790223216265354-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:14:04.294541 5725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:14:04.295969 5725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:14:04.304284 5731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:04.304383 5725 server_base.cc:1034] running on GCE node
W20260924 04:14:04.304268 5730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:04.304296 5733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:04.305161 5725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:14:04.306221 5725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:14:04.307376 5725 hybrid_clock.cc:648] HybridClock initialized: now 1790223244307347 us; error 40 us; skew 500 ppm
I20260924 04:14:04.309279 5725 webserver.cc:460] Webserver started at http://127.0.98.62:33857/ using document root <none> and password file <none>
I20260924 04:14:04.309820 5725 fs_manager.cc:359] Metadata directory not provided
I20260924 04:14:04.309888 5725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:14:04.310091 5725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:14:04.312186 5725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790223216265354-392-0/minicluster-data/master-0/data/instance:
uuid: "d7329df8b284456fa6b2a23ff41d1f1b"
format_stamp: "Formatted at 2026-09-24 04:14:04 on dist-test-slave-p401"
server_key: "a9a4609f8345fac3c0c59b13ee3ba3f2"
server_key_iv: "62ca474898a1d649b2b3fc3b5d8e695d"
server_key_version: "encryptionkey@0"
I20260924 04:14:04.312705 5725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790223216265354-392-0/minicluster-data/master-0/wal/instance:
uuid: "d7329df8b284456fa6b2a23ff41d1f1b"
format_stamp: "Formatted at 2026-09-24 04:14:04 on dist-test-slave-p401"
server_key: "a9a4609f8345fac3c0c59b13ee3ba3f2"
server_key_iv: "62ca474898a1d649b2b3fc3b5d8e695d"
server_key_version: "encryptionkey@0"
I20260924 04:14:04.316377 5725 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:14:04.318718 5739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:04.320003 5725 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:14:04.320112 5725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790223216265354-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790223216265354-392-0/minicluster-data/master-0/wal
uuid: "d7329df8b284456fa6b2a23ff41d1f1b"
format_stamp: "Formatted at 2026-09-24 04:14:04 on dist-test-slave-p401"
server_key: "a9a4609f8345fac3c0c59b13ee3ba3f2"
server_key_iv: "62ca474898a1d649b2b3fc3b5d8e695d"
server_key_version: "encryptionkey@0"
I20260924 04:14:04.320227 5725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790223216265354-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790223216265354-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790223216265354-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:14:04.338711 5725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:14:04.339460 5725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:14:04.339638 5725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:14:04.346460 5725 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45915
I20260924 04:14:04.346506 5791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45915 every 8 connection(s)
I20260924 04:14:04.347456 5725 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:14:04.350742 5792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:14:04.350829 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 5725
I20260924 04:14:04.350956 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790223216265354-392-0/minicluster-data/master-0/wal/instance
I20260924 04:14:04.351221 392 external_mini_cluster.cc:1428] Setting key 838e4ab5a96fd0e9eaefb139c41189d8
I20260924 04:14:04.357591 5792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7329df8b284456fa6b2a23ff41d1f1b: Bootstrap starting.
I20260924 04:14:04.360268 5792 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7329df8b284456fa6b2a23ff41d1f1b: Neither blocks nor log segments found. Creating new log.
I20260924 04:14:04.360726 5795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:04.352625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38554 (local address 127.0.98.62:45915)
0924 04:14:04.353253 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:04.353281 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:04.353326 (+ 45us) server_negotiation.cc:407] Connection header received
0924 04:14:04.354043 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:04.354061 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:04.354482 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:04.354784 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:04.355133 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:04.356176 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:04.357357 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:04.357913 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:04.358040 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:04.358923 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:04.358928 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:04.358942 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:14:04.359383 (+ 441us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:04.359441 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:14:04.359462 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:04.359537 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:04.359686 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 04:14:04.359954 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":107,"threads_started":1}
I20260924 04:14:04.361024 5792 log.cc:824] T 00000000000000000000000000000000 P d7329df8b284456fa6b2a23ff41d1f1b: Log is configured to *not* fsync() on all Append() calls
I20260924 04:14:04.363051 5792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7329df8b284456fa6b2a23ff41d1f1b: No bootstrap required, opened a new log
I20260924 04:14:04.363602 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:14:04.366765 5792 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7329df8b284456fa6b2a23ff41d1f1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7329df8b284456fa6b2a23ff41d1f1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45915 } }
I20260924 04:14:04.367009 5792 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7329df8b284456fa6b2a23ff41d1f1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:14:04.367081 5792 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7329df8b284456fa6b2a23ff41d1f1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7329df8b284456fa6b2a23ff41d1f1b, State: Initialized, Role: FOLLOWER
I20260924 04:14:04.367558 5792 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7329df8b284456fa6b2a23ff41d1f1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7329df8b284456fa6b2a23ff41d1f1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45915 } }
I20260924 04:14:04.367698 5792 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7329df8b284456fa6b2a23ff41d1f1b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:14:04.367766 5792 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7329df8b284456fa6b2a23ff41d1f1b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:14:04.367862 5792 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7329df8b284456fa6b2a23ff41d1f1b [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:14:04.368813 5792 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7329df8b284456fa6b2a23ff41d1f1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7329df8b284456fa6b2a23ff41d1f1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45915 } }
I20260924 04:14:04.369131 5792 leader_election.cc:302] T 00000000000000000000000000000000 P d7329df8b284456fa6b2a23ff41d1f1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7329df8b284456fa6b2a23ff41d1f1b; no voters:
I20260924 04:14:04.369422 5792 leader_election.cc:288] T 00000000000000000000000000000000 P d7329df8b284456fa6b2a23ff41d1f1b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:14:04.369560 5797 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7329df8b284456fa6b2a23ff41d1f1b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:14:04.369789 5797 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7329df8b284456fa6b2a23ff41d1f1b [term 1 LEADER]: Becoming Leader. State: Replica: d7329df8b284456fa6b2a23ff41d1f1b, State: Running, Role: LEADER
I20260924 04:14:04.370095 5797 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7329df8b284456fa6b2a23ff41d1f1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7329df8b284456fa6b2a23ff41d1f1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45915 } }
I20260924 04:14:04.370674 5792 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7329df8b284456fa6b2a23ff41d1f1b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:14:04.371855 5799 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7329df8b284456fa6b2a23ff41d1f1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7329df8b284456fa6b2a23ff41d1f1b. Latest consensus state: current_term: 1 leader_uuid: "d7329df8b284456fa6b2a23ff41d1f1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7329df8b284456fa6b2a23ff41d1f1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45915 } } }
I20260924 04:14:04.371876 5798 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7329df8b284456fa6b2a23ff41d1f1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7329df8b284456fa6b2a23ff41d1f1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7329df8b284456fa6b2a23ff41d1f1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45915 } } }
I20260924 04:14:04.371982 5799 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7329df8b284456fa6b2a23ff41d1f1b [sys.catalog]: This master's current role is: LEADER
I20260924 04:14:04.371981 5798 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7329df8b284456fa6b2a23ff41d1f1b [sys.catalog]: This master's current role is: LEADER
I20260924 04:14:04.372335 5807 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:14:04.375308 5807 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:14:04.380700 5807 catalog_manager.cc:1347] Generated new cluster ID: 0bc66cfb488341239f6b7844f16edcc4
I20260924 04:14:04.380829 5807 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:14:04.400749 5807 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:14:04.401736 5807 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:14:04.410902 5807 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7329df8b284456fa6b2a23ff41d1f1b: Generated new TSK 0
I20260924 04:14:04.411695 5807 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:14:04.432735 5795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:04.427082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38564 (local address 127.0.98.62:45915)
0924 04:14:04.427215 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:04.427219 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:04.427391 (+ 172us) server_negotiation.cc:407] Connection header received
0924 04:14:04.427539 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:04.427542 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:04.427599 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:04.427719 (+ 120us) server_negotiation.cc:226] Negotiated authn=JWT
0924 04:14:04.427897 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:04.428736 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:04.429546 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:04.429882 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:04.429976 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:04.430221 (+ 245us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 04:14:04.432342 (+ 2121us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43861/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 04:14:04.432583 (+ 241us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38564: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43861/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":46}
W20260924 04:14:04.432808 5795 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38564: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43861/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260924 04:14:04.434352 5821 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:04.426992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45915 (local address 127.0.0.1:38564)
0924 04:14:04.427312 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:04.427325 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:04.427426 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:04.427708 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:04.427710 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:04.427715 (+ 5us) client_negotiation.cc:189] Negotiated authn=JWT
0924 04:14:04.427807 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:04.427811 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:04.428826 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:04.428833 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:04.429448 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:04.429455 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:04.429960 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:04.429979 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:04.430034 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:04.430055 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 04:14:04.432528 (+ 2473us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43861/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 04:14:04.434186 (+ 1658us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:45915: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43861/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":1426,"thread_start_us":103,"threads_started":1}
I20260924 04:14:04.435055 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 5725
E20260924 04:14:04.445423 5822 webserver.cc:533] Webserver: sslize failed:
E20260924 04:14:04.445508 5822 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-24T04:14:04Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (308 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-24T04:14:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:14:04Z Disabled control of system clock
I20260924 04:14:04.482569 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790223216265354-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790223216265354-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790223216265354-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33513
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33513
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790223216265354-392-0/testchainca.cert with env {}
W20260924 04:14:04.590296 5826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:14:04.590575 5826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:14:04.590631 5826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:14:04.594656 5826 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:14:04.594729 5826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:14:04.594751 5826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:14:04.594771 5826 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:14:04.594794 5826 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 04:14:04.594823 5826 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 04:14:04.594851 5826 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260924 04:14:04.600410 5826 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790223216265354-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790223216265354-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33513
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790223216265354-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790223216265354-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:14:04.601508 5826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:14:04.602512 5826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:14:04.610680 5831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:04.610733 5834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:04.610733 5832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:04.610868 5826 server_base.cc:1034] running on GCE node
I20260924 04:14:04.611656 5826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:14:04.612766 5826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:14:04.613945 5826 hybrid_clock.cc:648] HybridClock initialized: now 1790223244613926 us; error 32 us; skew 500 ppm
I20260924 04:14:04.615751 5826 webserver.cc:460] Webserver started at http://127.0.98.62:46597/ using document root <none> and password file <none>
I20260924 04:14:04.616318 5826 fs_manager.cc:359] Metadata directory not provided
I20260924 04:14:04.616386 5826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:14:04.616565 5826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:14:04.618621 5826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790223216265354-392-0/minicluster-data/master-0/data/instance:
uuid: "67fdbe5f58794ea09d90ff030284d108"
format_stamp: "Formatted at 2026-09-24 04:14:04 on dist-test-slave-p401"
server_key: "421778332fa0dc7aff5194debc57d711"
server_key_iv: "89c9ce1e6bb2dd32858694364c66a267"
server_key_version: "encryptionkey@0"
I20260924 04:14:04.619156 5826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790223216265354-392-0/minicluster-data/master-0/wal/instance:
uuid: "67fdbe5f58794ea09d90ff030284d108"
format_stamp: "Formatted at 2026-09-24 04:14:04 on dist-test-slave-p401"
server_key: "421778332fa0dc7aff5194debc57d711"
server_key_iv: "89c9ce1e6bb2dd32858694364c66a267"
server_key_version: "encryptionkey@0"
I20260924 04:14:04.622670 5826 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:14:04.624936 5840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:04.626128 5826 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 04:14:04.626233 5826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790223216265354-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790223216265354-392-0/minicluster-data/master-0/wal
uuid: "67fdbe5f58794ea09d90ff030284d108"
format_stamp: "Formatted at 2026-09-24 04:14:04 on dist-test-slave-p401"
server_key: "421778332fa0dc7aff5194debc57d711"
server_key_iv: "89c9ce1e6bb2dd32858694364c66a267"
server_key_version: "encryptionkey@0"
I20260924 04:14:04.626328 5826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790223216265354-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790223216265354-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790223216265354-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:14:04.646622 5826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:14:04.647367 5826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:14:04.647521 5826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:14:04.653823 5826 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33513
I20260924 04:14:04.653854 5892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33513 every 8 connection(s)
I20260924 04:14:04.654745 5826 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:14:04.657758 5893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:14:04.657913 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 5826
I20260924 04:14:04.658022 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790223216265354-392-0/minicluster-data/master-0/wal/instance
I20260924 04:14:04.658303 392 external_mini_cluster.cc:1428] Setting key 683d5219058af650d57bbef4967dfd3b
I20260924 04:14:04.664397 5893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67fdbe5f58794ea09d90ff030284d108: Bootstrap starting.
I20260924 04:14:04.666654 5893 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67fdbe5f58794ea09d90ff030284d108: Neither blocks nor log segments found. Creating new log.
I20260924 04:14:04.667210 5896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:04.659738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56610 (local address 127.0.98.62:33513)
0924 04:14:04.660382 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:04.660400 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:04.660436 (+ 36us) server_negotiation.cc:407] Connection header received
0924 04:14:04.661080 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:04.661097 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:04.661492 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:04.661805 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:04.662239 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:04.663207 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:04.664076 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:04.664551 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:04.664697 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:04.665482 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:04.665487 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:04.665499 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:14:04.665872 (+ 373us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:04.665929 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:14:04.665948 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:04.666037 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:04.666182 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 04:14:04.666438 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":126,"threads_started":1}
I20260924 04:14:04.667464 5893 log.cc:824] T 00000000000000000000000000000000 P 67fdbe5f58794ea09d90ff030284d108: Log is configured to *not* fsync() on all Append() calls
I20260924 04:14:04.669536 5893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67fdbe5f58794ea09d90ff030284d108: No bootstrap required, opened a new log
I20260924 04:14:04.670008 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:14:04.673350 5893 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67fdbe5f58794ea09d90ff030284d108 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67fdbe5f58794ea09d90ff030284d108" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33513 } }
I20260924 04:14:04.673599 5893 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67fdbe5f58794ea09d90ff030284d108 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:14:04.673664 5893 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67fdbe5f58794ea09d90ff030284d108 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67fdbe5f58794ea09d90ff030284d108, State: Initialized, Role: FOLLOWER
I20260924 04:14:04.674079 5893 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67fdbe5f58794ea09d90ff030284d108 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67fdbe5f58794ea09d90ff030284d108" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33513 } }
I20260924 04:14:04.674220 5893 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67fdbe5f58794ea09d90ff030284d108 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:14:04.674291 5893 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67fdbe5f58794ea09d90ff030284d108 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:14:04.674394 5893 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67fdbe5f58794ea09d90ff030284d108 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:14:04.675419 5893 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67fdbe5f58794ea09d90ff030284d108 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67fdbe5f58794ea09d90ff030284d108" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33513 } }
I20260924 04:14:04.675750 5893 leader_election.cc:302] T 00000000000000000000000000000000 P 67fdbe5f58794ea09d90ff030284d108 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67fdbe5f58794ea09d90ff030284d108; no voters:
I20260924 04:14:04.676066 5893 leader_election.cc:288] T 00000000000000000000000000000000 P 67fdbe5f58794ea09d90ff030284d108 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:14:04.676180 5898 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67fdbe5f58794ea09d90ff030284d108 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:14:04.676391 5898 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67fdbe5f58794ea09d90ff030284d108 [term 1 LEADER]: Becoming Leader. State: Replica: 67fdbe5f58794ea09d90ff030284d108, State: Running, Role: LEADER
I20260924 04:14:04.676667 5898 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67fdbe5f58794ea09d90ff030284d108 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67fdbe5f58794ea09d90ff030284d108" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33513 } }
I20260924 04:14:04.677281 5893 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67fdbe5f58794ea09d90ff030284d108 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:14:04.678193 5900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67fdbe5f58794ea09d90ff030284d108 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67fdbe5f58794ea09d90ff030284d108. Latest consensus state: current_term: 1 leader_uuid: "67fdbe5f58794ea09d90ff030284d108" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67fdbe5f58794ea09d90ff030284d108" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33513 } } }
I20260924 04:14:04.678323 5900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67fdbe5f58794ea09d90ff030284d108 [sys.catalog]: This master's current role is: LEADER
I20260924 04:14:04.678603 5899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67fdbe5f58794ea09d90ff030284d108 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67fdbe5f58794ea09d90ff030284d108" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67fdbe5f58794ea09d90ff030284d108" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33513 } } }
I20260924 04:14:04.678689 5899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67fdbe5f58794ea09d90ff030284d108 [sys.catalog]: This master's current role is: LEADER
I20260924 04:14:04.678771 5907 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:14:04.681891 5907 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:14:04.687664 5907 catalog_manager.cc:1347] Generated new cluster ID: e647d2ebd2024d8cabaaeeb2f8d15dda
I20260924 04:14:04.687734 5907 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:14:04.717710 5907 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:14:04.719017 5907 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:14:04.724675 5907 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67fdbe5f58794ea09d90ff030284d108: Generated new TSK 0
I20260924 04:14:04.725658 5907 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:14:04.834558 5896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:04.742053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56618 (local address 127.0.98.62:33513)
0924 04:14:04.742183 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:04.742188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:04.742352 (+ 164us) server_negotiation.cc:407] Connection header received
0924 04:14:04.742510 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:04.742513 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:04.742559 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:04.742651 (+ 92us) server_negotiation.cc:226] Negotiated authn=JWT
0924 04:14:04.742878 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:04.743447 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:04.744302 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:04.744518 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:04.744603 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:04.744901 (+ 298us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 04:14:04.834092 (+ 89191us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 04:14:04.834400 (+ 308us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56618: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":49}
W20260924 04:14:04.834657 5896 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56618: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260924 04:14:04.834698 5922 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:04.741971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33513 (local address 127.0.0.1:56618)
0924 04:14:04.742276 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:04.742288 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:04.742389 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:04.742659 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:04.742662 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:04.742670 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0924 04:14:04.742785 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:04.742791 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:04.743575 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:04.743579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:04.744197 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:04.744206 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:04.744608 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:04.744631 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:04.744725 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:04.744747 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 04:14:04.834346 (+ 89599us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 04:14:04.834526 (+ 180us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:33513: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":89,"threads_started":1}
I20260924 04:14:04.835421 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 5826
2026-09-24T04:14:04Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (378 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (120 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (122 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:14:05 dist-test-slave-p401 krb5kdc[5934](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:14:05 dist-test-slave-p401 krb5kdc[5934](info): set up 2 sockets
Sep 24 04:14:05 dist-test-slave-p401 krb5kdc[5934](info): commencing operation
krb5kdc: starting...
W20260924 04:14:06.023516 392 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/krb5kdc/test-user.keytab.
Sep 24 04:14:06 dist-test-slave-p401 krb5kdc[5934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:14:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:14:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:14:06.140690 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34227
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34227
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:14:06.248549 5947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:14:06.248816 5947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:14:06.248867 5947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:14:06.252852 5947 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:14:06.252911 5947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:14:06.252938 5947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:14:06.252957 5947 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:14:06.252976 5947 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:14:06.258177 5947 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:14:06.259223 5947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:14:06.260247 5947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:14:06.268442 5955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:06.268427 5953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:06.268734 5947 server_base.cc:1034] running on GCE node
W20260924 04:14:06.268428 5952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:06.269403 5947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:14:06.270519 5947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:14:06.271668 5947 hybrid_clock.cc:648] HybridClock initialized: now 1790223246271640 us; error 43 us; skew 500 ppm
Sep 24 04:14:06 dist-test-slave-p401 krb5kdc[5934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:14:06.274289 5947 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:14:06.275128 5947 webserver.cc:460] Webserver started at http://127.0.98.62:41295/ using document root <none> and password file <none>
I20260924 04:14:06.275663 5947 fs_manager.cc:359] Metadata directory not provided
I20260924 04:14:06.275730 5947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:14:06.275909 5947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:14:06.277854 5947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/master-0/data/instance:
uuid: "d38fb98e7ac74196ba6e99dfb1ac4f7a"
format_stamp: "Formatted at 2026-09-24 04:14:06 on dist-test-slave-p401"
server_key: "e2751acb1acb433d010d396e0c198482"
server_key_iv: "2a3c3063244e13b1d412ab6caeb9097f"
server_key_version: "encryptionkey@0"
I20260924 04:14:06.278331 5947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/master-0/wal/instance:
uuid: "d38fb98e7ac74196ba6e99dfb1ac4f7a"
format_stamp: "Formatted at 2026-09-24 04:14:06 on dist-test-slave-p401"
server_key: "e2751acb1acb433d010d396e0c198482"
server_key_iv: "2a3c3063244e13b1d412ab6caeb9097f"
server_key_version: "encryptionkey@0"
I20260924 04:14:06.281872 5947 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260924 04:14:06.284224 5962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:06.285465 5947 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:14:06.285575 5947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/master-0/wal
uuid: "d38fb98e7ac74196ba6e99dfb1ac4f7a"
format_stamp: "Formatted at 2026-09-24 04:14:06 on dist-test-slave-p401"
server_key: "e2751acb1acb433d010d396e0c198482"
server_key_iv: "2a3c3063244e13b1d412ab6caeb9097f"
server_key_version: "encryptionkey@0"
I20260924 04:14:06.285667 5947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:14:06.309581 5947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:14:06.312769 5947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:14:06.312942 5947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:14:06.319497 5947 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34227
I20260924 04:14:06.319810 6014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34227 every 8 connection(s)
I20260924 04:14:06.320452 5947 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:14:06.323112 6015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:14:06.326730 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 5947
I20260924 04:14:06.326860 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/master-0/wal/instance
I20260924 04:14:06.327152 392 external_mini_cluster.cc:1428] Setting key c85f30e130e169172b2713442633aea8
I20260924 04:14:06.328845 6015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d38fb98e7ac74196ba6e99dfb1ac4f7a: Bootstrap starting.
I20260924 04:14:06.330927 6015 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d38fb98e7ac74196ba6e99dfb1ac4f7a: Neither blocks nor log segments found. Creating new log.
I20260924 04:14:06.331805 6015 log.cc:824] T 00000000000000000000000000000000 P d38fb98e7ac74196ba6e99dfb1ac4f7a: Log is configured to *not* fsync() on all Append() calls
I20260924 04:14:06.333693 6015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d38fb98e7ac74196ba6e99dfb1ac4f7a: No bootstrap required, opened a new log
Sep 24 04:14:06 dist-test-slave-p401 krb5kdc[5934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:14:06.337355 6015 raft_consensus.cc:348] T 00000000000000000000000000000000 P d38fb98e7ac74196ba6e99dfb1ac4f7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d38fb98e7ac74196ba6e99dfb1ac4f7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34227 } }
I20260924 04:14:06.337649 6015 raft_consensus.cc:374] T 00000000000000000000000000000000 P d38fb98e7ac74196ba6e99dfb1ac4f7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:14:06.337718 6015 raft_consensus.cc:729] T 00000000000000000000000000000000 P d38fb98e7ac74196ba6e99dfb1ac4f7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d38fb98e7ac74196ba6e99dfb1ac4f7a, State: Initialized, Role: FOLLOWER
I20260924 04:14:06.338192 6015 consensus_queue.cc:260] T 00000000000000000000000000000000 P d38fb98e7ac74196ba6e99dfb1ac4f7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d38fb98e7ac74196ba6e99dfb1ac4f7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34227 } }
I20260924 04:14:06.338356 6015 raft_consensus.cc:388] T 00000000000000000000000000000000 P d38fb98e7ac74196ba6e99dfb1ac4f7a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:14:06.338428 6015 raft_consensus.cc:482] T 00000000000000000000000000000000 P d38fb98e7ac74196ba6e99dfb1ac4f7a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:14:06.338523 6015 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d38fb98e7ac74196ba6e99dfb1ac4f7a [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:14:06.339545 6015 raft_consensus.cc:504] T 00000000000000000000000000000000 P d38fb98e7ac74196ba6e99dfb1ac4f7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d38fb98e7ac74196ba6e99dfb1ac4f7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34227 } }
I20260924 04:14:06.339912 6015 leader_election.cc:302] T 00000000000000000000000000000000 P d38fb98e7ac74196ba6e99dfb1ac4f7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d38fb98e7ac74196ba6e99dfb1ac4f7a; no voters:
I20260924 04:14:06.340242 6015 leader_election.cc:288] T 00000000000000000000000000000000 P d38fb98e7ac74196ba6e99dfb1ac4f7a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:14:06.340364 6020 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d38fb98e7ac74196ba6e99dfb1ac4f7a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:14:06.340616 6020 raft_consensus.cc:686] T 00000000000000000000000000000000 P d38fb98e7ac74196ba6e99dfb1ac4f7a [term 1 LEADER]: Becoming Leader. State: Replica: d38fb98e7ac74196ba6e99dfb1ac4f7a, State: Running, Role: LEADER
I20260924 04:14:06.340952 6020 consensus_queue.cc:237] T 00000000000000000000000000000000 P d38fb98e7ac74196ba6e99dfb1ac4f7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d38fb98e7ac74196ba6e99dfb1ac4f7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34227 } }
I20260924 04:14:06.341521 6018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:06.328512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37660 (local address 127.0.98.62:34227)
0924 04:14:06.328938 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:06.328957 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:06.328998 (+ 41us) server_negotiation.cc:407] Connection header received
0924 04:14:06.329617 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:06.329635 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:06.329959 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:06.330273 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:06.331104 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:06.332443 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:06.333303 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:06.333964 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:06.334118 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:06.336374 (+ 2256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:06.336380 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:06.336397 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:06.336427 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:06.338281 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:06.338767 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:06.338773 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:06.338778 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:06.338916 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:06.339204 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:06.339207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:06.339209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:06.339735 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:06.339923 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:06.340270 (+ 347us) server_negotiation.cc:299] Negotiation successful
0924 04:14:06.340570 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":125,"threads_started":1}
I20260924 04:14:06.341549 6015 sys_catalog.cc:564] T 00000000000000000000000000000000 P d38fb98e7ac74196ba6e99dfb1ac4f7a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:14:06.342047 6020 sys_catalog.cc:455] T 00000000000000000000000000000000 P d38fb98e7ac74196ba6e99dfb1ac4f7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d38fb98e7ac74196ba6e99dfb1ac4f7a. Latest consensus state: current_term: 1 leader_uuid: "d38fb98e7ac74196ba6e99dfb1ac4f7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d38fb98e7ac74196ba6e99dfb1ac4f7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34227 } } }
I20260924 04:14:06.342218 6020 sys_catalog.cc:458] T 00000000000000000000000000000000 P d38fb98e7ac74196ba6e99dfb1ac4f7a [sys.catalog]: This master's current role is: LEADER
I20260924 04:14:06.342599 6027 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:14:06.343185 6022 sys_catalog.cc:455] T 00000000000000000000000000000000 P d38fb98e7ac74196ba6e99dfb1ac4f7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d38fb98e7ac74196ba6e99dfb1ac4f7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d38fb98e7ac74196ba6e99dfb1ac4f7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34227 } } }
I20260924 04:14:06.343335 6022 sys_catalog.cc:458] T 00000000000000000000000000000000 P d38fb98e7ac74196ba6e99dfb1ac4f7a [sys.catalog]: This master's current role is: LEADER
I20260924 04:14:06.346140 6027 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:14:06.352312 6027 catalog_manager.cc:1347] Generated new cluster ID: e9e617762e5a439ba6a80c686b891aa3
I20260924 04:14:06.352366 6027 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:14:06.369244 6027 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:14:06.370177 6027 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:14:06.382898 6027 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d38fb98e7ac74196ba6e99dfb1ac4f7a: Generated new TSK 0
I20260924 04:14:06.383641 6027 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:14:06.418006 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34227
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:14:06.525996 6041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:14:06.526242 6041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:14:06.526283 6041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:14:06.530155 6041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:14:06.530208 6041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:14:06.530261 6041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:14:06.535290 6041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:14:06.536180 6041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:14:06.537058 6041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:14:06.546077 6047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:06.546083 6046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:06.546145 6041 server_base.cc:1034] running on GCE node
W20260924 04:14:06.546094 6049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:06.546614 6041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:14:06.547254 6041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:14:06.548431 6041 hybrid_clock.cc:648] HybridClock initialized: now 1790223246548409 us; error 39 us; skew 500 ppm
Sep 24 04:14:06 dist-test-slave-p401 krb5kdc[5934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:14:06.555751 6041 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:14:06.556646 6041 webserver.cc:460] Webserver started at http://127.0.98.1:42175/ using document root <none> and password file <none>
I20260924 04:14:06.557199 6041 fs_manager.cc:359] Metadata directory not provided
I20260924 04:14:06.557263 6041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:14:06.557451 6041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:14:06.559489 6041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/ts-0/data/instance:
uuid: "855756b3b4ff48878e38a32e468aa264"
format_stamp: "Formatted at 2026-09-24 04:14:06 on dist-test-slave-p401"
server_key: "b218b4d617397620304220ce83d332e2"
server_key_iv: "17af20989164f73a549cc102a01087e0"
server_key_version: "encryptionkey@0"
I20260924 04:14:06.560020 6041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/ts-0/wal/instance:
uuid: "855756b3b4ff48878e38a32e468aa264"
format_stamp: "Formatted at 2026-09-24 04:14:06 on dist-test-slave-p401"
server_key: "b218b4d617397620304220ce83d332e2"
server_key_iv: "17af20989164f73a549cc102a01087e0"
server_key_version: "encryptionkey@0"
I20260924 04:14:06.563529 6041 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:14:06.566047 6056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:06.567323 6041 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:14:06.567421 6041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/ts-0/wal
uuid: "855756b3b4ff48878e38a32e468aa264"
format_stamp: "Formatted at 2026-09-24 04:14:06 on dist-test-slave-p401"
server_key: "b218b4d617397620304220ce83d332e2"
server_key_iv: "17af20989164f73a549cc102a01087e0"
server_key_version: "encryptionkey@0"
I20260924 04:14:06.567515 6041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:14:06.578702 6041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:14:06.581537 6041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:14:06.581705 6041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:14:06.582242 6041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:14:06.583026 6041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:14:06.583072 6041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:06.583124 6041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:14:06.583151 6041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:06.591581 6041 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45589
I20260924 04:14:06.591602 6169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45589 every 8 connection(s)
I20260924 04:14:06.592463 6041 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:14:06.593657 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 6041
I20260924 04:14:06.593779 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:14:06.594087 392 external_mini_cluster.cc:1428] Setting key 98329efc3d135c0a1a680ae4a9f918c8
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Sep 24 04:14:06 dist-test-slave-p401 krb5kdc[5934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:14:06.606307 6018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:06.594288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48985 (local address 127.0.98.62:34227)
0924 04:14:06.594405 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:06.594408 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:06.595400 (+ 992us) server_negotiation.cc:407] Connection header received
0924 04:14:06.596862 (+ 1462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:06.596865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:06.596928 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:06.596998 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:06.598216 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:06.598751 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:06.599943 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:06.600165 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:06.600258 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:06.602710 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:06.602714 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:06.602717 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:06.602742 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:06.604161 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:06.604743 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:06.604748 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:06.604750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:06.604847 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:06.605165 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:06.605169 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:06.605170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:06.605414 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:06.605523 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:06.606015 (+ 492us) server_negotiation.cc:299] Negotiation successful
0924 04:14:06.606139 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 04:14:06.607098 6172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:06.594757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34227 (local address 127.0.98.1:48985)
0924 04:14:06.595223 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:06.595303 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:06.596628 (+ 1325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:06.597182 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:06.597191 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:06.597551 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:06.598070 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:06.598085 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:06.598864 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:06.598868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:06.599827 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:06.599837 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:06.600306 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:06.600310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:06.600466 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:06.601050 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:06.601066 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:06.602553 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:14:06.604258 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:06.604264 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:06.604279 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:06.604610 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:06.604944 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:06.604947 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:06.604948 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:06.605078 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:06.605534 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:14:06.605539 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:14:06.605767 (+ 228us) client_negotiation.cc:769] Sending connection context
0924 04:14:06.605956 (+ 189us) client_negotiation.cc:240] Negotiation successful
0924 04:14:06.606275 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":118,"threads_started":1}
I20260924 04:14:06.608364 6170 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34227
I20260924 04:14:06.608649 6170 heartbeater.cc:464] Registering TS with master...
I20260924 04:14:06.609264 6170 heartbeater.cc:511] Master 127.0.98.62:34227 requested a full tablet report, sending...
I20260924 04:14:06.610843 5979 ts_manager.cc:194] Registered new tserver with Master: 855756b3b4ff48878e38a32e468aa264 (127.0.98.1:45589)
I20260924 04:14:06.611971 5979 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48985
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:14:06.620499 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34227
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:14:06.731676 6175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:14:06.732129 6175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:14:06.732232 6175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:14:06.737197 6175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:14:06.737262 6175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:14:06.737321 6175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:14:06.742717 6175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:14:06.743793 6175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:14:06.744737 6175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:14:06.754325 6180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:06.754354 6181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:06.754347 6183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:06.754428 6175 server_base.cc:1034] running on GCE node
I20260924 04:14:06.754976 6175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:14:06.755622 6175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:14:06.756767 6175 hybrid_clock.cc:648] HybridClock initialized: now 1790223246756736 us; error 48 us; skew 500 ppm
Sep 24 04:14:06 dist-test-slave-p401 krb5kdc[5934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:14:06.759348 6175 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:14:06.760185 6175 webserver.cc:460] Webserver started at http://127.0.98.2:33359/ using document root <none> and password file <none>
I20260924 04:14:06.760691 6175 fs_manager.cc:359] Metadata directory not provided
I20260924 04:14:06.760747 6175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:14:06.760921 6175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:14:06.762931 6175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/ts-1/data/instance:
uuid: "4da63eb4ef614aa78823a457afa23c9c"
format_stamp: "Formatted at 2026-09-24 04:14:06 on dist-test-slave-p401"
server_key: "a68a75d6aa9c66f99bb3749264d4151b"
server_key_iv: "25894d1a146abd8844746fd993c6e674"
server_key_version: "encryptionkey@0"
I20260924 04:14:06.763801 6175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4da63eb4ef614aa78823a457afa23c9c"
format_stamp: "Formatted at 2026-09-24 04:14:06 on dist-test-slave-p401"
server_key: "a68a75d6aa9c66f99bb3749264d4151b"
server_key_iv: "25894d1a146abd8844746fd993c6e674"
server_key_version: "encryptionkey@0"
I20260924 04:14:06.767765 6175 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 04:14:06.770253 6190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:06.771621 6175 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:14:06.771773 6175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/ts-1/wal
uuid: "4da63eb4ef614aa78823a457afa23c9c"
format_stamp: "Formatted at 2026-09-24 04:14:06 on dist-test-slave-p401"
server_key: "a68a75d6aa9c66f99bb3749264d4151b"
server_key_iv: "25894d1a146abd8844746fd993c6e674"
server_key_version: "encryptionkey@0"
I20260924 04:14:06.771888 6175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:14:06.790040 6175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:14:06.793309 6175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:14:06.793493 6175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:14:06.794094 6175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:14:06.794911 6175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:14:06.794960 6175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:06.795011 6175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:14:06.795039 6175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:06.803375 6175 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34939
I20260924 04:14:06.803406 6303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34939 every 8 connection(s)
I20260924 04:14:06.804239 6175 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:14:06.806396 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 6175
I20260924 04:14:06.806495 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:14:06.806787 392 external_mini_cluster.cc:1428] Setting key 8ca05ffc80b64cd3b1995eb84efe3f31
Sep 24 04:14:06 dist-test-slave-p401 krb5kdc[5934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 04:14:06.818615 6018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:06.806029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41965 (local address 127.0.98.62:34227)
0924 04:14:06.806161 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:06.806166 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:06.807106 (+ 940us) server_negotiation.cc:407] Connection header received
0924 04:14:06.808712 (+ 1606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:06.808716 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:06.808765 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:06.808843 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:06.810118 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:06.810641 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:06.812237 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:06.812451 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:06.812529 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:06.815130 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:06.815134 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:06.815137 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:06.815166 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:06.816588 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:06.817109 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:06.817112 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:06.817114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:06.817189 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:06.817506 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:06.817509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:06.817511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:06.817681 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:06.817760 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:06.818280 (+ 520us) server_negotiation.cc:299] Negotiation successful
0924 04:14:06.818440 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 04:14:06.819576 6306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:06.806406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34227 (local address 127.0.98.2:41965)
0924 04:14:06.806934 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:06.806991 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:06.808480 (+ 1489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:06.808999 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:06.809005 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:06.809352 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:06.809970 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:06.809991 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:06.810755 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:06.810758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:06.812105 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:06.812117 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:06.812595 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:06.812599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:06.812795 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:06.813382 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:06.813404 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:06.814961 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:14:06.816743 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:06.816748 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:06.816760 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:06.817000 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:06.817271 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:06.817274 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:06.817276 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:06.817418 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:06.817773 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:14:06.817780 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:14:06.818046 (+ 266us) client_negotiation.cc:769] Sending connection context
0924 04:14:06.818227 (+ 181us) client_negotiation.cc:240] Negotiation successful
0924 04:14:06.818545 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":118,"threads_started":1}
I20260924 04:14:06.820823 6304 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34227
I20260924 04:14:06.821046 6304 heartbeater.cc:464] Registering TS with master...
I20260924 04:14:06.821502 6304 heartbeater.cc:511] Master 127.0.98.62:34227 requested a full tablet report, sending...
I20260924 04:14:06.822525 5979 ts_manager.cc:194] Registered new tserver with Master: 4da63eb4ef614aa78823a457afa23c9c (127.0.98.2:34939)
I20260924 04:14:06.823309 5979 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41965
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:14:06.836202 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34227
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:14:06.944743 6309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:14:06.945006 6309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:14:06.945050 6309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:14:06.949035 6309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:14:06.949095 6309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:14:06.949154 6309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:14:06.954362 6309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:14:06.955430 6309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:14:06.956413 6309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:14:06.966049 6317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:06.966077 6314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:06.966109 6315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:06.966076 6309 server_base.cc:1034] running on GCE node
I20260924 04:14:06.966667 6309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:14:06.967386 6309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:14:06.968542 6309 hybrid_clock.cc:648] HybridClock initialized: now 1790223246968522 us; error 38 us; skew 500 ppm
Sep 24 04:14:06 dist-test-slave-p401 krb5kdc[5934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:14:06.971307 6309 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:14:06.972244 6309 webserver.cc:460] Webserver started at http://127.0.98.3:41563/ using document root <none> and password file <none>
I20260924 04:14:06.972766 6309 fs_manager.cc:359] Metadata directory not provided
I20260924 04:14:06.972829 6309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:14:06.972990 6309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:14:06.975016 6309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/ts-2/data/instance:
uuid: "e938de434c2841aa9e664d286828c034"
format_stamp: "Formatted at 2026-09-24 04:14:06 on dist-test-slave-p401"
server_key: "c1a8149317155695a1e1423a2c35fe74"
server_key_iv: "0d68af5c7e77079cddaccdab64dd3cc5"
server_key_version: "encryptionkey@0"
I20260924 04:14:06.975648 6309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e938de434c2841aa9e664d286828c034"
format_stamp: "Formatted at 2026-09-24 04:14:06 on dist-test-slave-p401"
server_key: "c1a8149317155695a1e1423a2c35fe74"
server_key_iv: "0d68af5c7e77079cddaccdab64dd3cc5"
server_key_version: "encryptionkey@0"
I20260924 04:14:06.979386 6309 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:14:06.981900 6324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:06.983132 6309 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:14:06.983276 6309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/ts-2/wal
uuid: "e938de434c2841aa9e664d286828c034"
format_stamp: "Formatted at 2026-09-24 04:14:06 on dist-test-slave-p401"
server_key: "c1a8149317155695a1e1423a2c35fe74"
server_key_iv: "0d68af5c7e77079cddaccdab64dd3cc5"
server_key_version: "encryptionkey@0"
I20260924 04:14:06.983402 6309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:14:07.003062 6309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:14:07.006079 6309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:14:07.006253 6309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:14:07.006855 6309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:14:07.007736 6309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:14:07.007783 6309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:07.007822 6309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:14:07.007849 6309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:07.016147 6309 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33769
I20260924 04:14:07.016168 6437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33769 every 8 connection(s)
I20260924 04:14:07.017031 6309 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:14:07.021965 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 6309
I20260924 04:14:07.022061 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:14:07.022306 392 external_mini_cluster.cc:1428] Setting key eb823eb93d3f7cbf8bcb6810061fd45e
Sep 24 04:14:07 dist-test-slave-p401 krb5kdc[5934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:14:07.030678 6018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:07.018768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57533 (local address 127.0.98.62:34227)
0924 04:14:07.018953 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:07.018956 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:07.019840 (+ 884us) server_negotiation.cc:407] Connection header received
0924 04:14:07.021230 (+ 1390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:07.021233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:07.021279 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:07.021351 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:07.022758 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:07.023399 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:07.024729 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:07.024944 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:07.025027 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:07.027518 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:07.027521 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:07.027523 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:07.027550 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:07.028823 (+ 1273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:07.029290 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:07.029293 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:07.029294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:07.029365 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:07.029617 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:07.029619 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:07.029620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:07.029820 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:07.029892 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:07.030399 (+ 507us) server_negotiation.cc:299] Negotiation successful
0924 04:14:07.030517 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260924 04:14:07.031471 6440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:07.019159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34227 (local address 127.0.98.3:57533)
0924 04:14:07.019655 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:07.019701 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:07.021035 (+ 1334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:07.021546 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:07.021553 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:07.022035 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:07.022597 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:07.022614 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:07.023523 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:07.023525 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:07.024615 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:07.024625 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:07.025082 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:07.025086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:07.025248 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:07.025786 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:07.025803 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:07.027340 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:14:07.028937 (+ 1597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:07.028942 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:07.028955 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:07.029210 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:07.029425 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:07.029427 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:07.029429 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:07.029556 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:07.029896 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:14:07.029901 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:14:07.030137 (+ 236us) client_negotiation.cc:769] Sending connection context
0924 04:14:07.030294 (+ 157us) client_negotiation.cc:240] Negotiation successful
0924 04:14:07.030556 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":117,"threads_started":1}
I20260924 04:14:07.032604 6438 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34227
I20260924 04:14:07.032851 6438 heartbeater.cc:464] Registering TS with master...
I20260924 04:14:07.033350 6438 heartbeater.cc:511] Master 127.0.98.62:34227 requested a full tablet report, sending...
I20260924 04:14:07.034318 5979 ts_manager.cc:194] Registered new tserver with Master: e938de434c2841aa9e664d286828c034 (127.0.98.3:33769)
I20260924 04:14:07.034942 5979 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57533
I20260924 04:14:07.035962 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/krb5kdc/test-admin.keytab.
Sep 24 04:14:07 dist-test-slave-p401 krb5kdc[5934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:14:07.052139 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260924 04:14:07.054653 6445 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:07.053439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39150 (local address 127.0.98.1:45589)
0924 04:14:07.053759 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:07.053769 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:07.053793 (+ 24us) server_negotiation.cc:407] Connection header received
0924 04:14:07.053843 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:07.053849 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:07.054030 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:07.054149 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:07.054513 (+ 364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39150: BlockingRecv error: recv got EOF from 127.0.0.1:39150 (error 108)
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":112,"threads_started":1}
W20260924 04:14:07.054936 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/krb5kdc/krb5cc)
Sep 24 04:14:07 dist-test-slave-p401 krb5kdc[5934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:14:07.055893 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 04:14:07 dist-test-slave-p401 krb5kdc[5934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:14:07.056766 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 04:14:07 dist-test-slave-p401 krb5kdc[5934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:14:07.065681 6445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:07.057617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39154 (local address 127.0.98.1:45589)
0924 04:14:07.057775 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:07.057778 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:07.057824 (+ 46us) server_negotiation.cc:407] Connection header received
0924 04:14:07.057953 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:07.057957 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:07.057998 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:07.058061 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:07.058563 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:07.059374 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:07.060175 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:07.060547 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:07.060637 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:07.062658 (+ 2021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:07.062670 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:07.062684 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:07.062708 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:07.064072 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:07.064443 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:07.064448 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:07.064449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:07.064523 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:07.064760 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:07.064763 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:07.064765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:07.064988 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:07.065206 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:07.065357 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 04:14:07.065499 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260924 04:14:07.069418 6445 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:07.068531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39166 (local address 127.0.98.1:45589)
0924 04:14:07.068637 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:07.068642 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:07.068716 (+ 74us) server_negotiation.cc:407] Connection header received
0924 04:14:07.068805 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:07.068808 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:07.068863 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:07.068934 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:07.069315 (+ 381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39166: BlockingRecv error: recv got EOF from 127.0.0.1:39166 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260924 04:14:07.069476 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/krb5kdc/krb5cc)
Sep 24 04:14:07 dist-test-slave-p401 krb5kdc[5934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:14:07.070638 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 04:14:07 dist-test-slave-p401 krb5kdc[5934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:14:07.071710 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 04:14:07 dist-test-slave-p401 krb5kdc[5934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:14:07.080777 6445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:07.072601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39178 (local address 127.0.98.1:45589)
0924 04:14:07.072726 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:07.072730 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:07.072932 (+ 202us) server_negotiation.cc:407] Connection header received
0924 04:14:07.073036 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:07.073039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:07.073097 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:07.073184 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:07.073655 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:07.074374 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:07.075419 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:07.075663 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:07.075737 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:07.077718 (+ 1981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:07.077721 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:07.077723 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:07.077743 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:07.079185 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:07.079637 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:07.079641 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:07.079643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:07.079726 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:07.079987 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:07.079990 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:07.079992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:07.080215 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:07.080327 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:07.080525 (+ 198us) server_negotiation.cc:299] Negotiation successful
0924 04:14:07.080624 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260924 04:14:07.083217 6445 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:07.082411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39192 (local address 127.0.98.1:45589)
0924 04:14:07.082547 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:07.082551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:07.082584 (+ 33us) server_negotiation.cc:407] Connection header received
0924 04:14:07.082697 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:07.082699 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:07.082739 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:07.082793 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:07.083123 (+ 330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39192: BlockingRecv error: recv got EOF from 127.0.0.1:39192 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260924 04:14:07.083320 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790223216265354-392-0/krb5kdc/krb5cc' of type FILE
Sep 24 04:14:07 dist-test-slave-p401 krb5kdc[5934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:14:07.084336 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 04:14:07 dist-test-slave-p401 krb5kdc[5934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:14:07.085203 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 04:14:07 dist-test-slave-p401 krb5kdc[5934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:14:07.093467 6445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:07.086002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39206 (local address 127.0.98.1:45589)
0924 04:14:07.086118 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:07.086123 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:07.086208 (+ 85us) server_negotiation.cc:407] Connection header received
0924 04:14:07.086339 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:07.086342 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:07.086383 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:07.086455 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:07.087006 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:07.087576 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:07.088348 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:07.088555 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:07.088651 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:07.090798 (+ 2147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:07.090801 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:07.090803 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:07.090824 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:07.092062 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:07.092438 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:07.092442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:07.092444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:07.092511 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:07.092742 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:07.092745 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:07.092747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:07.092903 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:07.093006 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:07.093220 (+ 214us) server_negotiation.cc:299] Negotiation successful
0924 04:14:07.093321 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 04:14:07.094411 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 6041
I20260924 04:14:07.098891 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 6175
I20260924 04:14:07.103511 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 6309
I20260924 04:14:07.109402 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 5947
2026-09-24T04:14:07Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2031 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:14:07 dist-test-slave-p401 krb5kdc[6458](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:14:07 dist-test-slave-p401 krb5kdc[6458](info): set up 2 sockets
Sep 24 04:14:07 dist-test-slave-p401 krb5kdc[6458](info): commencing operation
krb5kdc: starting...
W20260924 04:14:08.063457 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/krb5kdc/test-user.keytab.
Sep 24 04:14:08 dist-test-slave-p401 krb5kdc[6458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:14:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:14:08Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260924 04:14:08.177768 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34999
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34999
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:14:08.284228 6471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:14:08.284500 6471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:14:08.284540 6471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:14:08.288471 6471 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:14:08.288528 6471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:14:08.288546 6471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:14:08.288561 6471 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:14:08.288575 6471 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:14:08.293817 6471 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:14:08.294872 6471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:14:08.295909 6471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:14:08.304054 6476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:08.304070 6471 server_base.cc:1034] running on GCE node
W20260924 04:14:08.304096 6479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:08.304108 6477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:08.304838 6471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:14:08.305946 6471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:14:08.307101 6471 hybrid_clock.cc:648] HybridClock initialized: now 1790223248307076 us; error 45 us; skew 500 ppm
Sep 24 04:14:08 dist-test-slave-p401 krb5kdc[6458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223248, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:14:08.309779 6471 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260924 04:14:08.310698 6471 webserver.cc:460] Webserver started at http://127.0.98.62:34981/ using document root <none> and password file <none>
I20260924 04:14:08.311214 6471 fs_manager.cc:359] Metadata directory not provided
I20260924 04:14:08.311334 6471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:14:08.311558 6471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:14:08.313500 6471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-0/data/instance:
uuid: "c0d01bc511ef45b399351d4e77c4c85d"
format_stamp: "Formatted at 2026-09-24 04:14:08 on dist-test-slave-p401"
server_key: "10a109a7b39042420aba7e1250375291"
server_key_iv: "fa37a7e86f740040251512962041599d"
server_key_version: "encryptionkey@0"
I20260924 04:14:08.314018 6471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-0/wal/instance:
uuid: "c0d01bc511ef45b399351d4e77c4c85d"
format_stamp: "Formatted at 2026-09-24 04:14:08 on dist-test-slave-p401"
server_key: "10a109a7b39042420aba7e1250375291"
server_key_iv: "fa37a7e86f740040251512962041599d"
server_key_version: "encryptionkey@0"
I20260924 04:14:08.317500 6471 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260924 04:14:08.319831 6486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:08.321040 6471 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:14:08.321147 6471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-0/wal
uuid: "c0d01bc511ef45b399351d4e77c4c85d"
format_stamp: "Formatted at 2026-09-24 04:14:08 on dist-test-slave-p401"
server_key: "10a109a7b39042420aba7e1250375291"
server_key_iv: "fa37a7e86f740040251512962041599d"
server_key_version: "encryptionkey@0"
I20260924 04:14:08.321240 6471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:14:08.346387 6471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:14:08.349594 6471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:14:08.349771 6471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:14:08.356928 6471 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34999
I20260924 04:14:08.357163 6538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34999 every 8 connection(s)
I20260924 04:14:08.357947 6471 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:14:08.360754 6539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:14:08.363751 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 6471
I20260924 04:14:08.363874 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-0/wal/instance
I20260924 04:14:08.364130 392 external_mini_cluster.cc:1428] Setting key 3a8b238d99ba6868209054387a1d78bb
I20260924 04:14:08.366537 6539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0d01bc511ef45b399351d4e77c4c85d: Bootstrap starting.
I20260924 04:14:08.369104 6539 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0d01bc511ef45b399351d4e77c4c85d: Neither blocks nor log segments found. Creating new log.
I20260924 04:14:08.369843 6539 log.cc:824] T 00000000000000000000000000000000 P c0d01bc511ef45b399351d4e77c4c85d: Log is configured to *not* fsync() on all Append() calls
I20260924 04:14:08.371836 6539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0d01bc511ef45b399351d4e77c4c85d: No bootstrap required, opened a new log
Sep 24 04:14:08 dist-test-slave-p401 krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:14:08.375221 6539 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0d01bc511ef45b399351d4e77c4c85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d01bc511ef45b399351d4e77c4c85d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } }
I20260924 04:14:08.375473 6539 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0d01bc511ef45b399351d4e77c4c85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:14:08.375522 6539 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0d01bc511ef45b399351d4e77c4c85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0d01bc511ef45b399351d4e77c4c85d, State: Initialized, Role: FOLLOWER
I20260924 04:14:08.375928 6539 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0d01bc511ef45b399351d4e77c4c85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d01bc511ef45b399351d4e77c4c85d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } }
I20260924 04:14:08.376049 6539 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0d01bc511ef45b399351d4e77c4c85d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:14:08.376114 6539 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0d01bc511ef45b399351d4e77c4c85d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:14:08.376215 6539 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0d01bc511ef45b399351d4e77c4c85d [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:14:08.377032 6539 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0d01bc511ef45b399351d4e77c4c85d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d01bc511ef45b399351d4e77c4c85d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } }
I20260924 04:14:08.377388 6539 leader_election.cc:302] T 00000000000000000000000000000000 P c0d01bc511ef45b399351d4e77c4c85d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0d01bc511ef45b399351d4e77c4c85d; no voters:
I20260924 04:14:08.377671 6539 leader_election.cc:288] T 00000000000000000000000000000000 P c0d01bc511ef45b399351d4e77c4c85d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:14:08.377767 6544 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0d01bc511ef45b399351d4e77c4c85d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:14:08.377962 6544 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0d01bc511ef45b399351d4e77c4c85d [term 1 LEADER]: Becoming Leader. State: Replica: c0d01bc511ef45b399351d4e77c4c85d, State: Running, Role: LEADER
I20260924 04:14:08.378257 6544 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0d01bc511ef45b399351d4e77c4c85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d01bc511ef45b399351d4e77c4c85d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } }
I20260924 04:14:08.378342 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:08.365492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44884 (local address 127.0.98.62:34999)
0924 04:14:08.365885 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:08.365900 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:08.365936 (+ 36us) server_negotiation.cc:407] Connection header received
0924 04:14:08.366532 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:08.366548 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:08.366914 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:08.367203 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:08.368119 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:08.369106 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:08.369960 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:08.370423 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:08.370574 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:08.373067 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:08.373073 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:08.373087 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:08.373113 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:08.375345 (+ 2232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:08.375824 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:08.375828 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:08.375832 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:08.375940 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:08.376211 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:08.376215 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:08.376216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:08.376628 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:08.376820 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:08.377071 (+ 251us) server_negotiation.cc:299] Negotiation successful
0924 04:14:08.377355 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":112,"threads_started":1}
I20260924 04:14:08.378831 6539 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0d01bc511ef45b399351d4e77c4c85d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:14:08.379830 6545 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0d01bc511ef45b399351d4e77c4c85d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0d01bc511ef45b399351d4e77c4c85d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d01bc511ef45b399351d4e77c4c85d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } } }
I20260924 04:14:08.379946 6545 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0d01bc511ef45b399351d4e77c4c85d [sys.catalog]: This master's current role is: LEADER
I20260924 04:14:08.380148 6545 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0d01bc511ef45b399351d4e77c4c85d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0d01bc511ef45b399351d4e77c4c85d. Latest consensus state: current_term: 1 leader_uuid: "c0d01bc511ef45b399351d4e77c4c85d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d01bc511ef45b399351d4e77c4c85d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } } }
I20260924 04:14:08.380223 6545 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0d01bc511ef45b399351d4e77c4c85d [sys.catalog]: This master's current role is: LEADER
W20260924 04:14:08.383507 6559 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c0d01bc511ef45b399351d4e77c4c85d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:14:08.383571 6559 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:14:08.383641 6553 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:14:08.384303 6553 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:14:08.389109 6553 catalog_manager.cc:1347] Generated new cluster ID: 4c5e55d2a7c949cf9287fac82ca19300
I20260924 04:14:08.389185 6553 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:14:08.396093 6553 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:14:08.397369 6553 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:14:08.412237 6553 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0d01bc511ef45b399351d4e77c4c85d: Generated new TSK 0
I20260924 04:14:08.412972 6553 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260924 04:14:08.446879 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34999
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:14:08.554723 6565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:14:08.555012 6565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:14:08.555073 6565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:14:08.559043 6565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:14:08.559108 6565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:14:08.559187 6565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:14:08.564486 6565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6565
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:14:08.565570 6565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:14:08.566567 6565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:14:08.575825 6570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:08.576010 6565 server_base.cc:1034] running on GCE node
W20260924 04:14:08.575871 6573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:08.575836 6571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:08.576452 6565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:14:08.577069 6565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:14:08.578223 6565 hybrid_clock.cc:648] HybridClock initialized: now 1790223248578196 us; error 36 us; skew 500 ppm
Sep 24 04:14:08 dist-test-slave-p401 krb5kdc[6458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223248, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:14:08.580910 6565 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 04:14:08.581760 6565 webserver.cc:460] Webserver started at http://127.0.98.1:34333/ using document root <none> and password file <none>
I20260924 04:14:08.582340 6565 fs_manager.cc:359] Metadata directory not provided
I20260924 04:14:08.582401 6565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:14:08.582604 6565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:14:08.584620 6565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-0/data/instance:
uuid: "34d2513e892f45b781f55ccdeae8ecd7"
format_stamp: "Formatted at 2026-09-24 04:14:08 on dist-test-slave-p401"
server_key: "b58974f3e198af0e933715bbaa45f9f0"
server_key_iv: "377cc1e49ba182e3bb5399b3d81860f3"
server_key_version: "encryptionkey@0"
I20260924 04:14:08.585139 6565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-0/wal/instance:
uuid: "34d2513e892f45b781f55ccdeae8ecd7"
format_stamp: "Formatted at 2026-09-24 04:14:08 on dist-test-slave-p401"
server_key: "b58974f3e198af0e933715bbaa45f9f0"
server_key_iv: "377cc1e49ba182e3bb5399b3d81860f3"
server_key_version: "encryptionkey@0"
I20260924 04:14:08.588615 6565 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:14:08.590808 6580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:08.592022 6565 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:14:08.592124 6565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-0/wal
uuid: "34d2513e892f45b781f55ccdeae8ecd7"
format_stamp: "Formatted at 2026-09-24 04:14:08 on dist-test-slave-p401"
server_key: "b58974f3e198af0e933715bbaa45f9f0"
server_key_iv: "377cc1e49ba182e3bb5399b3d81860f3"
server_key_version: "encryptionkey@0"
I20260924 04:14:08.592226 6565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:14:08.617353 6565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:14:08.620168 6565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:14:08.620333 6565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:14:08.620904 6565 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:14:08.621704 6565 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:14:08.621749 6565 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:08.621814 6565 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:14:08.621845 6565 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:08.630057 6565 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36843
I20260924 04:14:08.630076 6693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36843 every 8 connection(s)
I20260924 04:14:08.630911 6565 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:14:08.632928 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 6565
I20260924 04:14:08.633050 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:14:08.633327 392 external_mini_cluster.cc:1428] Setting key 9fa35ed9cbb28524b91d3f91806fd3da
Sep 24 04:14:08 dist-test-slave-p401 krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223248, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260924 04:14:08.644366 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:08.632612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46363 (local address 127.0.98.62:34999)
0924 04:14:08.632748 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:08.632753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:08.633609 (+ 856us) server_negotiation.cc:407] Connection header received
0924 04:14:08.635022 (+ 1413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:08.635025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:08.635071 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:08.635142 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:08.636558 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:08.637346 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:08.638489 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:08.638844 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:08.638947 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:08.641152 (+ 2205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:08.641157 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:08.641160 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:08.641185 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:08.642467 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:08.642933 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:08.642937 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:08.642939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:08.643030 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:08.643364 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:08.643367 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:08.643368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:08.643545 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:08.643642 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:08.644064 (+ 422us) server_negotiation.cc:299] Negotiation successful
0924 04:14:08.644191 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 04:14:08.645107 6696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:08.632979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34999 (local address 127.0.98.1:46363)
0924 04:14:08.633449 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:08.633502 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:08.634785 (+ 1283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:08.635350 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:08.635359 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:08.635807 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:08.636427 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:08.636443 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:08.637458 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:08.637460 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:08.638380 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:08.638389 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:08.638973 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:08.638975 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:08.639109 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:08.639646 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:08.639662 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:08.640995 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:14:08.642569 (+ 1574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:08.642573 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:08.642585 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:08.642816 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:08.643127 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:08.643129 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:08.643130 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:08.643291 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:08.643619 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:14:08.643626 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:14:08.643873 (+ 247us) client_negotiation.cc:769] Sending connection context
0924 04:14:08.644019 (+ 146us) client_negotiation.cc:240] Negotiation successful
0924 04:14:08.644284 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":124,"threads_started":1}
I20260924 04:14:08.646227 6694 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34999
I20260924 04:14:08.646446 6694 heartbeater.cc:464] Registering TS with master...
I20260924 04:14:08.646900 6694 heartbeater.cc:511] Master 127.0.98.62:34999 requested a full tablet report, sending...
I20260924 04:14:08.648259 6503 ts_manager.cc:194] Registered new tserver with Master: 34d2513e892f45b781f55ccdeae8ecd7 (127.0.98.1:36843)
I20260924 04:14:08.649327 6503 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46363
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260924 04:14:08.658490 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34999
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:14:08.767025 6699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:14:08.767318 6699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:14:08.767359 6699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:14:08.771324 6699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:14:08.771404 6699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:14:08.771467 6699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:14:08.776713 6699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:14:08.777719 6699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:14:08.778661 6699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:14:08.788144 6705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:08.788170 6704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:08.788147 6707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:08.788400 6699 server_base.cc:1034] running on GCE node
I20260924 04:14:08.788899 6699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:14:08.789549 6699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:14:08.790706 6699 hybrid_clock.cc:648] HybridClock initialized: now 1790223248790677 us; error 40 us; skew 500 ppm
Sep 24 04:14:08 dist-test-slave-p401 krb5kdc[6458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223248, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:14:08.793293 6699 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 04:14:08.794129 6699 webserver.cc:460] Webserver started at http://127.0.98.2:40225/ using document root <none> and password file <none>
I20260924 04:14:08.794672 6699 fs_manager.cc:359] Metadata directory not provided
I20260924 04:14:08.794734 6699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:14:08.794911 6699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:14:08.796959 6699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-1/data/instance:
uuid: "12f50fe1de024fe1b72357e102c83015"
format_stamp: "Formatted at 2026-09-24 04:14:08 on dist-test-slave-p401"
server_key: "3dff6e2b3ca7eac3fe48e6665007753a"
server_key_iv: "f0d5455bca9f26941ec41bc6de4f8470"
server_key_version: "encryptionkey@0"
I20260924 04:14:08.797488 6699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-1/wal/instance:
uuid: "12f50fe1de024fe1b72357e102c83015"
format_stamp: "Formatted at 2026-09-24 04:14:08 on dist-test-slave-p401"
server_key: "3dff6e2b3ca7eac3fe48e6665007753a"
server_key_iv: "f0d5455bca9f26941ec41bc6de4f8470"
server_key_version: "encryptionkey@0"
I20260924 04:14:08.800984 6699 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260924 04:14:08.803186 6714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:08.804373 6699 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 04:14:08.804472 6699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-1/wal
uuid: "12f50fe1de024fe1b72357e102c83015"
format_stamp: "Formatted at 2026-09-24 04:14:08 on dist-test-slave-p401"
server_key: "3dff6e2b3ca7eac3fe48e6665007753a"
server_key_iv: "f0d5455bca9f26941ec41bc6de4f8470"
server_key_version: "encryptionkey@0"
I20260924 04:14:08.804561 6699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:14:08.814564 6699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:14:08.817155 6699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:14:08.817320 6699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:14:08.817859 6699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:14:08.818601 6699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:14:08.818646 6699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:08.818696 6699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:14:08.818724 6699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:08.827195 6699 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33567
I20260924 04:14:08.827265 6827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33567 every 8 connection(s)
I20260924 04:14:08.828085 6699 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:14:08.833997 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 6699
I20260924 04:14:08.834115 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:14:08.834432 392 external_mini_cluster.cc:1428] Setting key 17d54401168dc0e9d462cc4c7a2d5f10
Sep 24 04:14:08 dist-test-slave-p401 krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223248, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:14:08.842193 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:08.829953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39745 (local address 127.0.98.62:34999)
0924 04:14:08.830106 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:08.830110 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:08.830994 (+ 884us) server_negotiation.cc:407] Connection header received
0924 04:14:08.832365 (+ 1371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:08.832368 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:08.832416 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:08.832491 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:08.833630 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:08.834191 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:08.835607 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:08.835892 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:08.835972 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:08.838731 (+ 2759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:08.838735 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:08.838737 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:08.838759 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:08.840136 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:08.840716 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:08.840719 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:08.840720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:08.840786 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:08.841130 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:08.841132 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:08.841133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:08.841295 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:08.841371 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:08.841980 (+ 609us) server_negotiation.cc:299] Negotiation successful
0924 04:14:08.842076 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 04:14:08.843458 6830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:08.830374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34999 (local address 127.0.98.2:39745)
0924 04:14:08.830834 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:08.830878 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:08.832154 (+ 1276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:08.832639 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:08.832645 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:08.832998 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:08.833501 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:08.833515 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:08.834312 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:08.834317 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:08.835514 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:08.835524 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:08.836035 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:08.836040 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:08.836227 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:08.836765 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:08.836783 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:08.838570 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:14:08.840252 (+ 1682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:08.840258 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:08.840276 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:08.840629 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:08.840862 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:08.840865 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:08.840867 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:08.841050 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:08.841397 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:14:08.841403 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:14:08.841728 (+ 325us) client_negotiation.cc:769] Sending connection context
0924 04:14:08.841964 (+ 236us) client_negotiation.cc:240] Negotiation successful
0924 04:14:08.842324 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":116,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:14:08.844529 6828 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34999
I20260924 04:14:08.844762 6828 heartbeater.cc:464] Registering TS with master...
I20260924 04:14:08.845259 6828 heartbeater.cc:511] Master 127.0.98.62:34999 requested a full tablet report, sending...
I20260924 04:14:08.846210 6503 ts_manager.cc:194] Registered new tserver with Master: 12f50fe1de024fe1b72357e102c83015 (127.0.98.2:33567)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
I20260924 04:14:08.846796 6503 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39745
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260924 04:14:08.860587 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34999
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:14:08.966239 6833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:14:08.966516 6833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:14:08.966558 6833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:14:08.970585 6833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:14:08.970655 6833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:14:08.970713 6833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:14:08.976074 6833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6833
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:14:08.977084 6833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:14:08.978047 6833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:14:08.987093 6839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:08.987185 6833 server_base.cc:1034] running on GCE node
W20260924 04:14:08.987116 6838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:08.987118 6841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:08.987772 6833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:14:08.988356 6833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:14:08.989527 6833 hybrid_clock.cc:648] HybridClock initialized: now 1790223248989507 us; error 28 us; skew 500 ppm
Sep 24 04:14:08 dist-test-slave-p401 krb5kdc[6458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223248, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:14:08.992121 6833 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 04:14:08.992902 6833 webserver.cc:460] Webserver started at http://127.0.98.3:35857/ using document root <none> and password file <none>
I20260924 04:14:08.993388 6833 fs_manager.cc:359] Metadata directory not provided
I20260924 04:14:08.993444 6833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:14:08.993607 6833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:14:08.995611 6833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-2/data/instance:
uuid: "7627c004b34c46e89a393afbfb63f656"
format_stamp: "Formatted at 2026-09-24 04:14:08 on dist-test-slave-p401"
server_key: "a73219c680d373174eb66acb01e4f96f"
server_key_iv: "80a10390833332016f00f4d794c50ebe"
server_key_version: "encryptionkey@0"
I20260924 04:14:08.996124 6833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7627c004b34c46e89a393afbfb63f656"
format_stamp: "Formatted at 2026-09-24 04:14:08 on dist-test-slave-p401"
server_key: "a73219c680d373174eb66acb01e4f96f"
server_key_iv: "80a10390833332016f00f4d794c50ebe"
server_key_version: "encryptionkey@0"
I20260924 04:14:08.999539 6833 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:14:09.001771 6848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:09.002930 6833 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:14:09.003029 6833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-2/wal
uuid: "7627c004b34c46e89a393afbfb63f656"
format_stamp: "Formatted at 2026-09-24 04:14:08 on dist-test-slave-p401"
server_key: "a73219c680d373174eb66acb01e4f96f"
server_key_iv: "80a10390833332016f00f4d794c50ebe"
server_key_version: "encryptionkey@0"
I20260924 04:14:09.003118 6833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:14:09.015472 6833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:14:09.018479 6833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:14:09.018646 6833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:14:09.019194 6833 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:14:09.020005 6833 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:14:09.020051 6833 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:09.020103 6833 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:14:09.020133 6833 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:09.028216 6833 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41945
I20260924 04:14:09.028240 6961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41945 every 8 connection(s)
I20260924 04:14:09.029065 6833 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:14:09.036214 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 6833
I20260924 04:14:09.036330 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:14:09.036592 392 external_mini_cluster.cc:1428] Setting key 8d1833ecaaf9593d649c40e12bced345
Sep 24 04:14:09 dist-test-slave-p401 krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223248, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:14:09.041703 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:09.030733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57971 (local address 127.0.98.62:34999)
0924 04:14:09.030865 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:09.030869 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:09.031718 (+ 849us) server_negotiation.cc:407] Connection header received
0924 04:14:09.032974 (+ 1256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:09.032978 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:09.033032 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:09.033105 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:09.034227 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.034737 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.035867 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.036138 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.036240 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:09.038682 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:09.038685 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:09.038687 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:09.038710 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:09.039901 (+ 1191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:09.040340 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:09.040344 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:09.040346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:09.040417 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:09.040710 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:09.040713 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:09.040715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:09.040892 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:09.040971 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:09.041440 (+ 469us) server_negotiation.cc:299] Negotiation successful
0924 04:14:09.041553 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 04:14:09.042480 6964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:09.031123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34999 (local address 127.0.98.3:57971)
0924 04:14:09.031573 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:09.031614 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:09.032772 (+ 1158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:09.033280 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:09.033287 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:09.033632 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:09.034090 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:09.034103 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.034846 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.034849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:09.035754 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:09.035762 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.036303 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.036307 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:09.036496 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:09.037037 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:09.037053 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:09.038539 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:14:09.039999 (+ 1460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:09.040005 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:09.040020 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:09.040255 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:09.040495 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:09.040498 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:09.040500 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:09.040629 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:09.040983 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:14:09.040990 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:14:09.041223 (+ 233us) client_negotiation.cc:769] Sending connection context
0924 04:14:09.041390 (+ 167us) client_negotiation.cc:240] Negotiation successful
0924 04:14:09.041700 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":135,"threads_started":1}
I20260924 04:14:09.043462 6962 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34999
I20260924 04:14:09.043665 6962 heartbeater.cc:464] Registering TS with master...
I20260924 04:14:09.044087 6962 heartbeater.cc:511] Master 127.0.98.62:34999 requested a full tablet report, sending...
I20260924 04:14:09.044971 6503 ts_manager.cc:194] Registered new tserver with Master: 7627c004b34c46e89a393afbfb63f656 (127.0.98.3:41945)
I20260924 04:14:09.045478 6503 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57971
I20260924 04:14:09.049890 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260924 04:14:09.074393 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45405
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34999,127.0.98.61:45405
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:14:09.182765 6967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:14:09.183035 6967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:14:09.183079 6967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:14:09.187039 6967 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:14:09.187101 6967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:14:09.187120 6967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:14:09.187135 6967 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:14:09.187152 6967 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:14:09.192432 6967 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34999,127.0.98.61:45405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
W20260924 04:14:09.192641 6967 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:34999,127.0.98.61:45405'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260924 04:14:09.193501 6967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:14:09.194494 6967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:14:09.202927 6975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:09.202976 6972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:09.203070 6973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:09.202978 6967 server_base.cc:1034] running on GCE node
I20260924 04:14:09.203781 6967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:14:09.204824 6967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:14:09.206002 6967 hybrid_clock.cc:648] HybridClock initialized: now 1790223249205973 us; error 39 us; skew 500 ppm
Sep 24 04:14:09 dist-test-slave-p401 krb5kdc[6458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223249, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:14:09.208818 6967 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260924 04:14:09.209694 6967 webserver.cc:460] Webserver started at http://127.0.98.61:33671/ using document root <none> and password file <none>
I20260924 04:14:09.210198 6967 fs_manager.cc:359] Metadata directory not provided
I20260924 04:14:09.210251 6967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:14:09.210417 6967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:14:09.212469 6967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-1/data/instance:
uuid: "15bf6683fb84414380dea83f7e7f93d8"
format_stamp: "Formatted at 2026-09-24 04:14:09 on dist-test-slave-p401"
server_key: "f757154a3231e3d37a995981c0ceed8c"
server_key_iv: "43e2804f74552b45ba66392048c518f9"
server_key_version: "encryptionkey@0"
I20260924 04:14:09.212987 6967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-1/wal/instance:
uuid: "15bf6683fb84414380dea83f7e7f93d8"
format_stamp: "Formatted at 2026-09-24 04:14:09 on dist-test-slave-p401"
server_key: "f757154a3231e3d37a995981c0ceed8c"
server_key_iv: "43e2804f74552b45ba66392048c518f9"
server_key_version: "encryptionkey@0"
I20260924 04:14:09.216609 6967 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:14:09.218895 6982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:09.220098 6967 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:14:09.220206 6967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-1/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-1/wal
uuid: "15bf6683fb84414380dea83f7e7f93d8"
format_stamp: "Formatted at 2026-09-24 04:14:09 on dist-test-slave-p401"
server_key: "f757154a3231e3d37a995981c0ceed8c"
server_key_iv: "43e2804f74552b45ba66392048c518f9"
server_key_version: "encryptionkey@0"
I20260924 04:14:09.220299 6967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:14:09.232264 6967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:14:09.234771 6967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:14:09.234937 6967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:14:09.241243 6967 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:45405
I20260924 04:14:09.241498 7034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:45405 every 8 connection(s)
I20260924 04:14:09.242165 6967 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-1/data/info.pb
I20260924 04:14:09.244819 7035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:14:09.248412 7035 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:14:09.249609 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 6967
Sep 24 04:14:09 dist-test-slave-p401 krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223249, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 24 04:14:09 dist-test-slave-p401 krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 04:14:09.261535 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:09.250158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44900 (local address 127.0.98.62:34999)
0924 04:14:09.250292 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:09.250297 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:09.251186 (+ 889us) server_negotiation.cc:407] Connection header received
0924 04:14:09.251954 (+ 768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:09.251957 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:09.252002 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:09.252092 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:09.253080 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.253877 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.255194 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.255549 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.255643 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:09.258050 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:09.258053 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:09.258055 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:09.258080 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:09.259480 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:09.260009 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:09.260013 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:09.260015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:09.260090 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:09.260392 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:09.260395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:09.260397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:09.260657 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:09.260774 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:09.261260 (+ 486us) server_negotiation.cc:299] Negotiation successful
0924 04:14:09.261375 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 04:14:09.262344 7038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:09.250610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34999 (local address 127.0.0.1:44900)
0924 04:14:09.251040 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:09.251086 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:09.251769 (+ 683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:09.252145 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:09.252150 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:09.252598 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:09.252940 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:09.252965 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.253987 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.253990 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:09.255082 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:09.255092 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.255681 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.255684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:09.255821 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:09.256374 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:09.256395 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:09.257917 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:14:09.259572 (+ 1655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:09.259580 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:09.259590 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:09.259923 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:09.260168 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:09.260170 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:09.260172 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:09.260310 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:09.260786 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:14:09.260793 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:14:09.261062 (+ 269us) client_negotiation.cc:769] Sending connection context
0924 04:14:09.261205 (+ 143us) client_negotiation.cc:240] Negotiation successful
0924 04:14:09.261513 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":23,"spinlock_wait_cycles":3840,"thread_start_us":109,"threads_started":1}
I20260924 04:14:09.262348 7039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:09.250611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54788 (local address 127.0.98.61:45405)
0924 04:14:09.251016 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:09.251031 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:09.251071 (+ 40us) server_negotiation.cc:407] Connection header received
0924 04:14:09.251760 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:09.251777 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:09.251999 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:09.252155 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:09.252968 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.254146 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.255209 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.255762 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.255847 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:09.258432 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:09.258437 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:09.258447 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:09.258466 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:09.259940 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:09.260311 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:09.260316 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:09.260317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:09.260385 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:09.260638 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:09.260641 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:09.260643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:09.261056 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:09.261223 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:09.261431 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 04:14:09.261560 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":297,"spinlock_wait_cycles":48256,"thread_start_us":163,"threads_started":1}
I20260924 04:14:09.263628 7035 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45405 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 04:14:09 dist-test-slave-p401 krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223249, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 04:14:09.270236 7038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:09.264077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45405 (local address 127.0.0.1:54796)
0924 04:14:09.264173 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:09.264186 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:09.264258 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:09.264443 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:09.264447 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:09.264579 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:09.264660 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:09.264665 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.265277 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.265279 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:09.265776 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:09.265781 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.266124 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.266126 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:09.266190 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:09.266527 (+ 337us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:09.266537 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:09.267615 (+ 1078us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:14:09.269217 (+ 1602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:09.269219 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:09.269221 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:09.269386 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:09.269578 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:09.269580 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:09.269581 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:09.269639 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:09.269889 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:14:09.269891 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:14:09.269954 (+ 63us) client_negotiation.cc:769] Sending connection context
0924 04:14:09.270020 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 04:14:09.270112 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 04:14:09.270265 7039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:09.264191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54796 (local address 127.0.98.61:45405)
0924 04:14:09.264268 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:09.264272 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:09.264307 (+ 35us) server_negotiation.cc:407] Connection header received
0924 04:14:09.264333 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:09.264334 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:09.264364 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:09.264413 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:09.264742 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.265208 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.265846 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.266057 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.266121 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:09.267726 (+ 1605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:09.267729 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:09.267731 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:09.267749 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:09.269144 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:09.269452 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:09.269454 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:09.269456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:09.269522 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:09.269693 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:09.269695 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:09.269696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:09.269831 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:09.269892 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:09.270049 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 04:14:09.270155 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":21}
I20260924 04:14:09.273169 7035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8: Bootstrap starting.
I20260924 04:14:09.275312 7035 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8: Neither blocks nor log segments found. Creating new log.
I20260924 04:14:09.275979 7035 log.cc:824] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8: Log is configured to *not* fsync() on all Append() calls
I20260924 04:14:09.277727 7035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8: No bootstrap required, opened a new log
I20260924 04:14:09.280781 7035 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0d01bc511ef45b399351d4e77c4c85d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } } peers { permanent_uuid: "15bf6683fb84414380dea83f7e7f93d8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45405 } }
I20260924 04:14:09.280970 7035 raft_consensus.cc:374] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:14:09.281019 7035 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15bf6683fb84414380dea83f7e7f93d8, State: Initialized, Role: FOLLOWER
I20260924 04:14:09.281417 7035 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0d01bc511ef45b399351d4e77c4c85d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } } peers { permanent_uuid: "15bf6683fb84414380dea83f7e7f93d8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45405 } }
I20260924 04:14:09.281826 7041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0d01bc511ef45b399351d4e77c4c85d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } } peers { permanent_uuid: "15bf6683fb84414380dea83f7e7f93d8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45405 } } }
I20260924 04:14:09.281932 7041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:14:09.282502 7035 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 04:14:09.286676 7052 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:14:09.286755 7052 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:14:09.288607 6967 master_runner.cc:428] Detected that this master 15bf6683fb84414380dea83f7e7f93d8 is joining an existing cluster
I20260924 04:14:09.288668 6967 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:45405
I20260924 04:14:09.297904 7058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:09.290738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34999 (local address 127.0.0.1:44912)
0924 04:14:09.291012 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:09.291026 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:09.291120 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:09.291452 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:09.291455 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:09.291679 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:09.291841 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:09.291849 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.292556 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.292559 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:09.293143 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:09.293150 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.293556 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.293559 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:09.293639 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:09.294079 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:09.294090 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:09.294438 (+ 348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:14:09.296328 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:09.296331 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:09.296333 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:09.296524 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:09.296794 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:09.296797 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:09.296799 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:09.296902 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:09.297352 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:14:09.297355 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:14:09.297450 (+ 95us) client_negotiation.cc:769] Sending connection context
0924 04:14:09.297577 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 04:14:09.297708 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":96,"threads_started":1}
I20260924 04:14:09.297926 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:09.290811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44912 (local address 127.0.98.62:34999)
0924 04:14:09.290941 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:09.290945 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:09.291090 (+ 145us) server_negotiation.cc:407] Connection header received
0924 04:14:09.291200 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:09.291203 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:09.291306 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:09.291396 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:09.291936 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.292462 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.293239 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.293458 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.293540 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:09.294566 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:09.294571 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:09.294574 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:09.294599 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:09.296228 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:09.296607 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:09.296611 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:09.296613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:09.296698 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:09.297012 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:09.297016 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:09.297017 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:09.297224 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:09.297288 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:09.297656 (+ 368us) server_negotiation.cc:299] Negotiation successful
0924 04:14:09.297768 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:14:09.299314 7039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:09.291161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54802 (local address 127.0.98.61:45405)
0924 04:14:09.291271 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:09.291275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:09.291729 (+ 454us) server_negotiation.cc:407] Connection header received
0924 04:14:09.291808 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:09.291811 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:09.291859 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:09.291932 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:09.292470 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.293207 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.294277 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.294584 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.294662 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:09.296194 (+ 1532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:09.296197 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:09.296200 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:09.296221 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:09.297549 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:09.297966 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:09.297970 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:09.297971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:09.298040 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:09.298540 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:09.298543 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:09.298545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:09.298730 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:09.298815 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:09.299056 (+ 241us) server_negotiation.cc:299] Negotiation successful
0924 04:14:09.299151 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 04:14:09.299767 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:09.291339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45405 (local address 127.0.0.1:54802)
0924 04:14:09.291639 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:09.291652 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:09.291738 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:09.292019 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:09.292022 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:09.292236 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:09.292356 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:09.292363 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.293321 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.293324 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:09.294167 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:09.294176 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.294666 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.294670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:09.294749 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:09.295726 (+ 977us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:09.295737 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:09.296083 (+ 346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:14:09.297651 (+ 1568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:09.297654 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:09.297656 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:09.297876 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:09.298364 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:09.298367 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:09.298369 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:09.298451 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:09.298855 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:14:09.298857 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:14:09.298924 (+ 67us) client_negotiation.cc:769] Sending connection context
0924 04:14:09.299509 (+ 585us) client_negotiation.cc:240] Negotiation successful
0924 04:14:09.299620 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":83,"threads_started":1}
Sep 24 04:14:09 dist-test-slave-p401 krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223248, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 04:14:09.310539 7060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:09.302288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45405 (local address 127.0.0.1:54806)
0924 04:14:09.302605 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:09.302653 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:09.302816 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:09.303095 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:09.303099 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:09.303471 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:09.303570 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:09.303576 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.304487 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.304489 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:09.305088 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:09.305095 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.305582 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.305585 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:09.305660 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:09.306153 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:09.306164 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:09.307556 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:14:09.309199 (+ 1643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:09.309207 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:09.309211 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:09.309427 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:09.309654 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:09.309656 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:09.309657 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:09.309721 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:09.310061 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:14:09.310067 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:14:09.310161 (+ 94us) client_negotiation.cc:769] Sending connection context
0924 04:14:09.310265 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 04:14:09.310417 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":85,"threads_started":1}
I20260924 04:14:09.310561 7039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:09.302354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54806 (local address 127.0.98.61:45405)
0924 04:14:09.302456 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:09.302460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:09.302717 (+ 257us) server_negotiation.cc:407] Connection header received
0924 04:14:09.302944 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:09.302947 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:09.302994 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:09.303069 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:09.303662 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.304405 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.305174 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.305493 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.305580 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:09.307715 (+ 2135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:09.307719 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:09.307722 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:09.307745 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:09.309111 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:09.309505 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:09.309508 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:09.309510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:09.309584 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:09.309785 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:09.309787 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:09.309789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:09.309979 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:09.310069 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:09.310321 (+ 252us) server_negotiation.cc:299] Negotiation successful
0924 04:14:09.310420 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 04:14:09.311816 6503 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:45405: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "15bf6683fb84414380dea83f7e7f93d8"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 45405
}
attrs {
promote: true
}
}
dest_uuid: "c0d01bc511ef45b399351d4e77c4c85d"
cas_config_opid_index: -1
I20260924 04:14:09.312376 6503 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0d01bc511ef45b399351d4e77c4c85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0d01bc511ef45b399351d4e77c4c85d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } } peers { permanent_uuid: "15bf6683fb84414380dea83f7e7f93d8" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45405 } attrs { promote: true } }
I20260924 04:14:09.313864 7062 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c0d01bc511ef45b399351d4e77c4c85d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 15bf6683fb84414380dea83f7e7f93d8 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0d01bc511ef45b399351d4e77c4c85d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } } peers { permanent_uuid: "15bf6683fb84414380dea83f7e7f93d8" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45405 } attrs { promote: true } } }
I20260924 04:14:09.314512 7009 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:14:09.315272 7062 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:45405
I20260924 04:14:09.315542 7062 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0d01bc511ef45b399351d4e77c4c85d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c0d01bc511ef45b399351d4e77c4c85d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0d01bc511ef45b399351d4e77c4c85d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } } peers { permanent_uuid: "15bf6683fb84414380dea83f7e7f93d8" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45405 } attrs { promote: true } } }
I20260924 04:14:09.315626 7062 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0d01bc511ef45b399351d4e77c4c85d [sys.catalog]: This master's current role is: LEADER
I20260924 04:14:09.315652 6967 master.cc:537] Master@127.0.98.61:45405 shutting down...
I20260924 04:14:09.315794 7009 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8 [term 1 FOLLOWER]: Refusing update from remote peer c0d01bc511ef45b399351d4e77c4c85d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260924 04:14:09.316610 7062 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c0d01bc511ef45b399351d4e77c4c85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "15bf6683fb84414380dea83f7e7f93d8" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45405 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 04:14:09.318773 6490 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260924 04:14:09.319703 6490 consensus_peers.cc:487] T 00000000000000000000000000000000 P c0d01bc511ef45b399351d4e77c4c85d -> Peer 15bf6683fb84414380dea83f7e7f93d8 (127.0.98.61:45405): Couldn't send request to peer 15bf6683fb84414380dea83f7e7f93d8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 04:14:09.320282 6967 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:14:09.320549 6967 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:14:09.320649 6967 tablet_replica.cc:330] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8: stopping tablet replica
I20260924 04:14:09.324816 6967 master.cc:552] Master@127.0.98.61:45405 shutdown complete.
I20260924 04:14:09.324868 6967 master_runner.cc:305] Clearing existing system tablet
I20260924 04:14:09.325091 6967 fs_manager.cc:359] Metadata directory not provided
I20260924 04:14:09.325153 6967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:14:09.326396 6967 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 04:14:09.327219 7067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:09.327502 6967 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:14:09.327621 6967 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 04:14:09.327687 6967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-1/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-1/wal
uuid: "15bf6683fb84414380dea83f7e7f93d8"
format_stamp: "Formatted at 2026-09-24 04:14:09 on dist-test-slave-p401"
server_key: "f757154a3231e3d37a995981c0ceed8c"
server_key_iv: "43e2804f74552b45ba66392048c518f9"
server_key_version: "encryptionkey@0"
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260924 04:14:09.329859 6967 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8: Deleting tablet data with delete state TABLET_DATA_DELETED
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
I20260924 04:14:09.332302 6967 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 04:14:09.332458 6967 log.cc:1196] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260924 04:14:09.332724 6967 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8: Deleting consensus metadata
I20260924 04:14:09.332878 6967 master_runner.cc:315] Copying system tablet from 127.0.98.62:34999
I20260924 04:14:09.334002 6967 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:34999
I20260924 04:14:09.342693 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:09.334654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44918 (local address 127.0.98.62:34999)
0924 04:14:09.334780 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:09.334785 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:09.335309 (+ 524us) server_negotiation.cc:407] Connection header received
0924 04:14:09.335397 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:09.335400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:09.335458 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:09.335549 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:09.336016 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.336856 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.337987 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.338354 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.338462 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:09.339756 (+ 1294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:09.339760 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:09.339762 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:09.339790 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:09.341149 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:09.341581 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:09.341584 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:09.341586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:09.341663 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:09.341901 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:09.341903 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:09.341904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:09.342093 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:09.342161 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:09.342423 (+ 262us) server_negotiation.cc:299] Negotiation successful
0924 04:14:09.342542 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 04:14:09.342710 7074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:09.334879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34999 (local address 127.0.0.1:44918)
0924 04:14:09.335145 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:09.335161 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:09.335306 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:09.335569 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:09.335572 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:09.335781 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:09.335903 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:09.335910 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.336964 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.336968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:09.337861 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:09.337871 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.338469 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.338473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:09.338565 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:09.339165 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:09.339180 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:09.339623 (+ 443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:14:09.341254 (+ 1631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:09.341258 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:09.341260 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:09.341498 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:09.341735 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:09.341738 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:09.341740 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:09.341829 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:09.342176 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:14:09.342179 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:14:09.342277 (+ 98us) client_negotiation.cc:769] Sending connection context
0924 04:14:09.342369 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 04:14:09.342508 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":98,"threads_started":1}
I20260924 04:14:09.343343 6523 tablet_copy_service.cc:140] P c0d01bc511ef45b399351d4e77c4c85d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 15bf6683fb84414380dea83f7e7f93d8 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:44918)
I20260924 04:14:09.343523 6523 tablet_copy_service.cc:161] P c0d01bc511ef45b399351d4e77c4c85d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 15bf6683fb84414380dea83f7e7f93d8 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:44918: session id = 15bf6683fb84414380dea83f7e7f93d8-00000000000000000000000000000000
I20260924 04:14:09.345088 6523 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c0d01bc511ef45b399351d4e77c4c85d: Tablet Copy: opened 0 blocks and 1 log segments
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260924 04:14:09.346487 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36165
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34999,127.0.98.61:45405,127.0.98.60:36165
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:14:09.347913 6967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 04:14:09.350236 6967 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8: tablet copy: Starting download of 0 data blocks...
I20260924 04:14:09.350448 6967 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8: tablet copy: Starting download of 1 WAL segments...
I20260924 04:14:09.352371 6967 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 04:14:09.354305 6523 tablet_copy_service.cc:342] P c0d01bc511ef45b399351d4e77c4c85d: Request end of tablet copy session 15bf6683fb84414380dea83f7e7f93d8-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:44918
I20260924 04:14:09.354446 6523 tablet_copy_service.cc:433] P c0d01bc511ef45b399351d4e77c4c85d: ending tablet copy session 15bf6683fb84414380dea83f7e7f93d8-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 15bf6683fb84414380dea83f7e7f93d8
I20260924 04:14:09.362399 6967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:14:09.364022 7076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:09.368177 7077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:09.368481 7079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:09.368609 6967 server_base.cc:1034] running on GCE node
I20260924 04:14:09.368981 6967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:14:09.369264 6967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:14:09.370393 6967 hybrid_clock.cc:648] HybridClock initialized: now 1790223249370381 us; error 26 us; skew 500 ppm
Sep 24 04:14:09 dist-test-slave-p401 krb5kdc[6458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223249, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:14:09.371958 6967 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260924 04:14:09.372390 6967 webserver.cc:460] Webserver started at http://127.0.98.61:40589/ using document root <none> and password file <none>
I20260924 04:14:09.372560 6967 fs_manager.cc:359] Metadata directory not provided
I20260924 04:14:09.372594 6967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:14:09.373888 6967 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 04:14:09.374732 7086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:09.374940 6967 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 04:14:09.375007 6967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-1/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-1/wal
uuid: "15bf6683fb84414380dea83f7e7f93d8"
format_stamp: "Formatted at 2026-09-24 04:14:09 on dist-test-slave-p401"
server_key: "f757154a3231e3d37a995981c0ceed8c"
server_key_iv: "43e2804f74552b45ba66392048c518f9"
server_key_version: "encryptionkey@0"
I20260924 04:14:09.375093 6967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:14:09.418437 6967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:14:09.420246 6967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:14:09.424625 6967 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:45405
I20260924 04:14:09.424664 7138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:45405 every 8 connection(s)
I20260924 04:14:09.425199 6967 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-1/data/info.pb
I20260924 04:14:09.426591 7139 sys_catalog.cc:263] Verifying existing consensus state
I20260924 04:14:09.427407 7139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8: Bootstrap starting.
I20260924 04:14:09.435247 7139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:14:09.435518 7139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8: Bootstrap complete.
I20260924 04:14:09.435982 7139 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0d01bc511ef45b399351d4e77c4c85d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } } peers { permanent_uuid: "15bf6683fb84414380dea83f7e7f93d8" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45405 } attrs { promote: true } }
I20260924 04:14:09.436086 7139 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 15bf6683fb84414380dea83f7e7f93d8, State: Initialized, Role: LEARNER
I20260924 04:14:09.436180 7139 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0d01bc511ef45b399351d4e77c4c85d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } } peers { permanent_uuid: "15bf6683fb84414380dea83f7e7f93d8" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45405 } attrs { promote: true } }
I20260924 04:14:09.436479 7142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0d01bc511ef45b399351d4e77c4c85d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } } peers { permanent_uuid: "15bf6683fb84414380dea83f7e7f93d8" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45405 } attrs { promote: true } } }
I20260924 04:14:09.436614 7139 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:14:09.436740 7142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8 [sys.catalog]: This master's current role is: LEARNER
I20260924 04:14:09.439520 7153 catalog_manager.cc:1259] Loaded cluster ID: 4c5e55d2a7c949cf9287fac82ca19300
I20260924 04:14:09.439631 7153 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8: loading cluster ID for follower catalog manager: success
I20260924 04:14:09.441152 7153 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8: acquiring CA information for follower catalog manager: success
I20260924 04:14:09.442261 7153 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 04:14:09.492437 7075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:14:09.492705 7075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:14:09.492744 7075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:14:09.496634 7075 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:14:09.496687 7075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:14:09.496706 7075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:14:09.496721 7075 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:14:09.496740 7075 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:14:09.501809 7075 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34999,127.0.98.61:45405,127.0.98.60:36165
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:14:09.502835 7075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:14:09.503861 7075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:14:09.511976 7075 server_base.cc:1034] running on GCE node
W20260924 04:14:09.512027 7158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:09.511976 7159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:09.512136 7161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:09.512694 7075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:14:09.513695 7075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:14:09.514845 7075 hybrid_clock.cc:648] HybridClock initialized: now 1790223249514830 us; error 31 us; skew 500 ppm
Sep 24 04:14:09 dist-test-slave-p401 krb5kdc[6458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223249, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:14:09.517306 7075 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260924 04:14:09.518131 7075 webserver.cc:460] Webserver started at http://127.0.98.60:36703/ using document root <none> and password file <none>
I20260924 04:14:09.518644 7075 fs_manager.cc:359] Metadata directory not provided
I20260924 04:14:09.518707 7075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:14:09.518885 7075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:14:09.520871 7075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-2/data/instance:
uuid: "86c8e5a65b994a6d91ae2f5ff4fc249d"
format_stamp: "Formatted at 2026-09-24 04:14:09 on dist-test-slave-p401"
server_key: "4b32af0b17e35421d72be9272ece1643"
server_key_iv: "4f2ab3397dd384661ae2d2317c45b605"
server_key_version: "encryptionkey@0"
I20260924 04:14:09.521399 7075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-2/wal/instance:
uuid: "86c8e5a65b994a6d91ae2f5ff4fc249d"
format_stamp: "Formatted at 2026-09-24 04:14:09 on dist-test-slave-p401"
server_key: "4b32af0b17e35421d72be9272ece1643"
server_key_iv: "4f2ab3397dd384661ae2d2317c45b605"
server_key_version: "encryptionkey@0"
I20260924 04:14:09.524958 7075 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260924 04:14:09.527220 7168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:09.528398 7075 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:14:09.528501 7075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-2/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-2/wal
uuid: "86c8e5a65b994a6d91ae2f5ff4fc249d"
format_stamp: "Formatted at 2026-09-24 04:14:09 on dist-test-slave-p401"
server_key: "4b32af0b17e35421d72be9272ece1643"
server_key_iv: "4f2ab3397dd384661ae2d2317c45b605"
server_key_version: "encryptionkey@0"
I20260924 04:14:09.528587 7075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:14:09.549868 7075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:14:09.552547 7075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:14:09.552718 7075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:14:09.559087 7075 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:36165
I20260924 04:14:09.559352 7220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:36165 every 8 connection(s)
I20260924 04:14:09.560038 7075 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-2/data/info.pb
I20260924 04:14:09.562654 7221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:14:09.563427 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 7075
I20260924 04:14:09.567566 7221 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 04:14:09 dist-test-slave-p401 krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 24 04:14:09 dist-test-slave-p401 krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223249, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:14:09.576683 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:09.564821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41654 (local address 127.0.98.60:36165)
0924 04:14:09.565224 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:09.565240 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:09.565277 (+ 37us) server_negotiation.cc:407] Connection header received
0924 04:14:09.565922 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:09.565937 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:09.566256 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:09.566558 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:09.567334 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.568353 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.569250 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.569897 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.570034 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:09.572291 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:09.572298 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:09.572310 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:09.572340 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:09.573876 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:09.574292 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:09.574298 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:09.574304 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:09.574424 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:09.574677 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:09.574680 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:09.574681 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:09.575061 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:09.575276 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:09.575591 (+ 315us) server_negotiation.cc:299] Negotiation successful
0924 04:14:09.575863 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":118,"threads_started":1}
I20260924 04:14:09.578547 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:09.569231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44928 (local address 127.0.98.62:34999)
0924 04:14:09.569354 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:09.569358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:09.569614 (+ 256us) server_negotiation.cc:407] Connection header received
0924 04:14:09.569724 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:09.569726 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:09.569768 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:09.569832 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:09.570525 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.571012 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.572045 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.572303 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.572423 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:09.574608 (+ 2185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:09.574612 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:09.574614 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:09.574637 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:09.576048 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:09.576527 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:09.576531 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:09.576532 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:09.576617 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:09.577762 (+ 1145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:09.577764 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:09.577766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:09.577940 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:09.578022 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:09.578331 (+ 309us) server_negotiation.cc:299] Negotiation successful
0924 04:14:09.578419 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 04:14:09.578616 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:09.569037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34999 (local address 127.0.0.1:44928)
0924 04:14:09.569494 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:09.569539 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:09.569647 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:09.569897 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:09.569902 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:09.570297 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:09.570407 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:09.570412 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.571123 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.571125 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:09.571922 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:09.571931 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.572411 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.572413 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:09.572494 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:09.573017 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:09.573027 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:09.574502 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:14:09.576140 (+ 1638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:09.576147 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:09.576154 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:09.576437 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:09.577549 (+ 1112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:09.577554 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:09.577556 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:09.577649 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:09.578037 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:14:09.578042 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:14:09.578163 (+ 121us) client_negotiation.cc:769] Sending connection context
0924 04:14:09.578293 (+ 130us) client_negotiation.cc:240] Negotiation successful
0924 04:14:09.578461 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":78,"threads_started":1}
I20260924 04:14:09.579620 7221 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45405 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 04:14:09 dist-test-slave-p401 krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223249, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 04:14:09.586711 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:09.580091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45405 (local address 127.0.0.1:54818)
0924 04:14:09.580201 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:09.580214 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:09.580292 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:09.580617 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:09.580619 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:09.580736 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:09.580865 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:09.580872 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.581693 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.581695 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:09.582216 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:09.582222 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.582591 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.582594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:09.582667 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:09.583066 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:09.583076 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:09.584170 (+ 1094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:14:09.585554 (+ 1384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:09.585557 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:09.585558 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:09.585718 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:09.585913 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:09.585915 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:09.585916 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:09.585975 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:09.586376 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:14:09.586378 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:14:09.586444 (+ 66us) client_negotiation.cc:769] Sending connection context
0924 04:14:09.586511 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 04:14:09.586600 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 04:14:09.586802 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:09.580190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54818 (local address 127.0.98.61:45405)
0924 04:14:09.580444 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:09.580447 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:09.580459 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:14:09.580493 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:09.580495 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:09.580531 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:09.580598 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:09.580955 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.581615 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.582290 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.582507 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.582573 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:09.584287 (+ 1714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:09.584290 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:09.584292 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:09.584314 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:09.585471 (+ 1157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:09.585786 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:09.585788 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:09.585789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:09.585855 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:09.586080 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:09.586083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:09.586084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:09.586290 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:09.586399 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:09.586549 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 04:14:09.586664 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":105,"threads_started":1}
I20260924 04:14:09.587466 7221 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36165 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 04:14:09 dist-test-slave-p401 krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223249, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 04:14:09.594383 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:09.587878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36165 (local address 127.0.0.1:41656)
0924 04:14:09.587999 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:09.588012 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:09.588069 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:09.588224 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:09.588226 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:09.588336 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:09.588397 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:09.588401 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.589330 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.589332 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:09.589817 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:09.589822 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.590210 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.590213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:09.590280 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:09.590661 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:09.590673 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:09.591831 (+ 1158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:14:09.593197 (+ 1366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:09.593199 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:09.593201 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:09.593366 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:09.593561 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:09.593563 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:09.593564 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:09.593631 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:09.593966 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:14:09.593969 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:14:09.594053 (+ 84us) client_negotiation.cc:769] Sending connection context
0924 04:14:09.594132 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 04:14:09.594239 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 04:14:09.594413 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:09.587956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41656 (local address 127.0.98.60:36165)
0924 04:14:09.588061 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:09.588064 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:09.588074 (+ 10us) server_negotiation.cc:407] Connection header received
0924 04:14:09.588120 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:09.588122 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:09.588153 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:09.588208 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:09.588468 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.589253 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.589889 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.590099 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.590163 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:09.591936 (+ 1773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:09.591939 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:09.591941 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:09.591960 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:09.593110 (+ 1150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:09.593433 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:09.593435 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:09.593436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:09.593495 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:09.593694 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:09.593696 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:09.593697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:09.593881 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:09.593949 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:09.594176 (+ 227us) server_negotiation.cc:299] Negotiation successful
0924 04:14:09.594288 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":40}
I20260924 04:14:09.597302 7221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d: Bootstrap starting.
I20260924 04:14:09.599521 7221 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d: Neither blocks nor log segments found. Creating new log.
I20260924 04:14:09.600232 7221 log.cc:824] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d: Log is configured to *not* fsync() on all Append() calls
I20260924 04:14:09.601972 7221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d: No bootstrap required, opened a new log
I20260924 04:14:09.605016 7221 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0d01bc511ef45b399351d4e77c4c85d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } } peers { permanent_uuid: "15bf6683fb84414380dea83f7e7f93d8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45405 } } peers { permanent_uuid: "86c8e5a65b994a6d91ae2f5ff4fc249d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36165 } }
I20260924 04:14:09.605206 7221 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:14:09.605243 7221 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86c8e5a65b994a6d91ae2f5ff4fc249d, State: Initialized, Role: FOLLOWER
I20260924 04:14:09.605609 7221 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0d01bc511ef45b399351d4e77c4c85d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } } peers { permanent_uuid: "15bf6683fb84414380dea83f7e7f93d8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45405 } } peers { permanent_uuid: "86c8e5a65b994a6d91ae2f5ff4fc249d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36165 } }
I20260924 04:14:09.606003 7227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0d01bc511ef45b399351d4e77c4c85d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } } peers { permanent_uuid: "15bf6683fb84414380dea83f7e7f93d8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45405 } } peers { permanent_uuid: "86c8e5a65b994a6d91ae2f5ff4fc249d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36165 } } }
I20260924 04:14:09.606114 7227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:14:09.606746 7221 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d [sys.catalog]: configured and running, proceeding with master startup.
W20260924 04:14:09.611096 7238 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:14:09.611171 7238 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:14:09.614091 7075 master_runner.cc:428] Detected that this master 86c8e5a65b994a6d91ae2f5ff4fc249d is joining an existing cluster
I20260924 04:14:09.614138 7075 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:36165
I20260924 04:14:09.629866 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 6565
I20260924 04:14:09.635298 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:36843
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34999
--webserver_port=34333
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:34999,127.0.98.61:45405
--tserver_master_addrs=127.0.98.62:34999,127.0.98.61:45405,127.0.98.60:36165 with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:14:09.640746 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:09.615968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34999 (local address 127.0.0.1:44938)
0924 04:14:09.616245 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:09.616257 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:09.616349 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:09.616681 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:09.616683 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:09.616914 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:09.617064 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:09.617070 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.617998 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.618000 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:09.618583 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:09.618591 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.619292 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.619296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:09.619386 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:09.619894 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:09.619909 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:09.620318 (+ 409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:14:09.621910 (+ 1592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:09.621913 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:09.621915 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:09.622140 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:09.622425 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:09.622428 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:09.622431 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:09.639443 (+ 17012us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:09.640089 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:14:09.640092 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:14:09.640236 (+ 144us) client_negotiation.cc:769] Sending connection context
0924 04:14:09.640348 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 04:14:09.640522 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":22,"spinlock_wait_cycles":3584,"thread_start_us":118,"threads_started":1}
I20260924 04:14:09.641310 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:09.616030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44938 (local address 127.0.98.62:34999)
0924 04:14:09.616142 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:09.616146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:09.616320 (+ 174us) server_negotiation.cc:407] Connection header received
0924 04:14:09.616455 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:09.616458 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:09.616507 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:09.616554 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:09.617181 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.617897 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.618690 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.619037 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.619135 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:09.620438 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:09.620441 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:09.620444 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:09.620469 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:09.621820 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:09.622246 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:09.622249 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:09.622251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:09.622333 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:09.639629 (+ 17296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:09.639634 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:09.639637 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:09.639926 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:09.640038 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:09.640992 (+ 954us) server_negotiation.cc:299] Negotiation successful
0924 04:14:09.641140 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 04:14:09.643961 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:09.616906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41658 (local address 127.0.98.60:36165)
0924 04:14:09.616998 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:09.617002 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:09.618868 (+ 1866us) server_negotiation.cc:407] Connection header received
0924 04:14:09.618996 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:09.618998 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:09.619028 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:09.619064 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:09.619902 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.620530 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.621538 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.621864 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.621970 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:09.640321 (+ 18351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:09.640325 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:09.640328 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:09.640352 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:09.642241 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:09.642694 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:09.642697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:09.642699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:09.642775 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:09.643051 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:09.643054 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:09.643056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:09.643280 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:09.643351 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:09.643641 (+ 290us) server_negotiation.cc:299] Negotiation successful
0924 04:14:09.643764 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":36}
I20260924 04:14:09.645270 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:09.617054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36165 (local address 127.0.0.1:41658)
0924 04:14:09.618806 (+ 1752us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:09.618817 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:09.618921 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:09.619152 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:09.619155 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:09.619380 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:09.619799 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:09.619807 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.620637 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.620640 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:09.621411 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:09.621422 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.621969 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.621972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:09.622141 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:09.622508 (+ 367us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:09.622519 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:09.639842 (+ 17323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:14:09.642357 (+ 2515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:09.642360 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:09.642363 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:09.642579 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:09.642885 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:09.642888 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:09.642890 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:09.642966 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:09.643407 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:14:09.643409 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:14:09.643487 (+ 78us) client_negotiation.cc:769] Sending connection context
0924 04:14:09.644265 (+ 778us) client_negotiation.cc:240] Negotiation successful
0924 04:14:09.645102 (+ 837us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1685,"mutex_wait_us":28,"thread_start_us":86,"threads_started":1}
I20260924 04:14:09.645394 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:09.616446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54822 (local address 127.0.98.61:45405)
0924 04:14:09.616550 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:09.616554 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:09.616897 (+ 343us) server_negotiation.cc:407] Connection header received
0924 04:14:09.617055 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:09.617058 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:09.617104 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:09.617151 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:09.617718 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.618491 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.619531 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.619869 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.619934 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:09.621141 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:09.621145 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:09.621149 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:09.621177 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:09.623274 (+ 2097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:09.641557 (+ 18283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:09.641562 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:09.641564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:09.641685 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:09.644361 (+ 2676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:09.644366 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:09.644368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:09.644618 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:09.644705 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:09.645071 (+ 366us) server_negotiation.cc:299] Negotiation successful
0924 04:14:09.645227 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 04:14:09.645925 7245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:09.616454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45405 (local address 127.0.0.1:54822)
0924 04:14:09.616824 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:09.616836 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:09.616957 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:09.617270 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:09.617273 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:09.617502 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:09.617601 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:09.617607 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.618599 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.618602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:09.619411 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:09.619420 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.620068 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.620072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:09.620162 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:09.620604 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:09.620617 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:09.621000 (+ 383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:14:09.623363 (+ 2363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:09.623365 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:09.623366 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:09.640168 (+ 16802us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:09.643766 (+ 3598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:09.643770 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:09.643772 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:09.644223 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:09.644783 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:14:09.644786 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:14:09.644888 (+ 102us) client_negotiation.cc:769] Sending connection context
0924 04:14:09.645593 (+ 705us) client_negotiation.cc:240] Negotiation successful
0924 04:14:09.645737 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"spinlock_wait_cycles":1408,"thread_start_us":80,"threads_started":1}
Sep 24 04:14:09 dist-test-slave-p401 krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223248, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 04:14:09.655896 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:09.646998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41670 (local address 127.0.98.60:36165)
0924 04:14:09.647099 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:09.647103 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:09.647115 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:14:09.647195 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:09.647198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:09.647275 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:09.647363 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:09.647908 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.648685 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.649782 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.650112 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.650200 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:09.652616 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:09.652620 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:09.652624 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:09.652649 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:09.654191 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:09.654634 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:09.654637 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:09.654639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:09.654722 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:09.654971 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:09.654974 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:09.654976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:09.655195 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:09.655316 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:09.655647 (+ 331us) server_negotiation.cc:299] Negotiation successful
0924 04:14:09.655759 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 04:14:09.655897 7060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:09.646542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36165 (local address 127.0.0.1:41670)
0924 04:14:09.646773 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:09.646788 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:09.646862 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:09.647487 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:09.647491 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:09.647683 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:09.647793 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:09.647801 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.648809 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.648813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:09.649672 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:09.649683 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.650216 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.650220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:09.650322 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:09.650792 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:09.650808 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:09.652468 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:14:09.654313 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:09.654317 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:09.654320 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:09.654546 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:09.654807 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:09.654810 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:09.654812 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:09.654890 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:09.655367 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:14:09.655371 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:14:09.655470 (+ 99us) client_negotiation.cc:769] Sending connection context
0924 04:14:09.655584 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 04:14:09.655725 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160}
I20260924 04:14:09.657338 6503 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:36165: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "86c8e5a65b994a6d91ae2f5ff4fc249d"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 36165
}
attrs {
promote: true
}
}
dest_uuid: "c0d01bc511ef45b399351d4e77c4c85d"
cas_config_opid_index: 5
I20260924 04:14:09.657979 6503 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0d01bc511ef45b399351d4e77c4c85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c0d01bc511ef45b399351d4e77c4c85d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } } peers { permanent_uuid: "15bf6683fb84414380dea83f7e7f93d8" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45405 } attrs { promote: true } } peers { permanent_uuid: "86c8e5a65b994a6d91ae2f5ff4fc249d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36165 } attrs { promote: true } }
I20260924 04:14:09.659034 7062 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c0d01bc511ef45b399351d4e77c4c85d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 86c8e5a65b994a6d91ae2f5ff4fc249d (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c0d01bc511ef45b399351d4e77c4c85d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } } peers { permanent_uuid: "15bf6683fb84414380dea83f7e7f93d8" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45405 } attrs { promote: true } } peers { permanent_uuid: "86c8e5a65b994a6d91ae2f5ff4fc249d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36165 } attrs { promote: true } } }
I20260924 04:14:09.660501 7062 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:36165
I20260924 04:14:09.660682 7062 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0d01bc511ef45b399351d4e77c4c85d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c0d01bc511ef45b399351d4e77c4c85d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c0d01bc511ef45b399351d4e77c4c85d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } } peers { permanent_uuid: "15bf6683fb84414380dea83f7e7f93d8" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45405 } attrs { promote: true } } peers { permanent_uuid: "86c8e5a65b994a6d91ae2f5ff4fc249d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36165 } attrs { promote: true } } }
I20260924 04:14:09.660790 7062 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0d01bc511ef45b399351d4e77c4c85d [sys.catalog]: This master's current role is: LEADER
I20260924 04:14:09.661648 7195 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:14:09.662969 7195 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d [term 1 FOLLOWER]: Refusing update from remote peer c0d01bc511ef45b399351d4e77c4c85d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 04:14:09.663594 7061 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c0d01bc511ef45b399351d4e77c4c85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "86c8e5a65b994a6d91ae2f5ff4fc249d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36165 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:14:09.665849 7060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:09.659626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45405 (local address 127.0.0.1:54826)
0924 04:14:09.661359 (+ 1733us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:09.661374 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:09.661454 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:09.661870 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:09.661873 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:09.661890 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:14:09.661977 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:09.661984 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.663012 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.663016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:09.664269 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:09.664280 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.665410 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.665414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:09.665514 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:09.665564 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 04:14:09.665644 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 04:14:09.665716 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1665}
I20260924 04:14:09.666291 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:09.661031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54826 (local address 127.0.98.61:45405)
0924 04:14:09.661145 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:09.661149 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:09.661540 (+ 391us) server_negotiation.cc:407] Connection header received
0924 04:14:09.661594 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:09.661597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:09.661658 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:09.661745 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:14:09.662100 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.662871 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.664420 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.665068 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.665191 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:09.665348 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:09.666048 (+ 700us) server_negotiation.cc:299] Negotiation successful
0924 04:14:09.666154 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:14:09.666707 7113 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8 [term 1 LEARNER]: Refusing update from remote peer c0d01bc511ef45b399351d4e77c4c85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 04:14:09.667115 7061 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c0d01bc511ef45b399351d4e77c4c85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "15bf6683fb84414380dea83f7e7f93d8" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45405 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 04:14:09.667726 7195 raft_consensus.cc:858] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0d01bc511ef45b399351d4e77c4c85d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } } peers { permanent_uuid: "15bf6683fb84414380dea83f7e7f93d8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45405 } } peers { permanent_uuid: "86c8e5a65b994a6d91ae2f5ff4fc249d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36165 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0d01bc511ef45b399351d4e77c4c85d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } } peers { permanent_uuid: "15bf6683fb84414380dea83f7e7f93d8" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45405 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c0d01bc511ef45b399351d4e77c4c85d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } } peers { permanent_uuid: "15bf6683fb84414380dea83f7e7f93d8" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45405 } attrs { promote: true } } peers { permanent_uuid: "86c8e5a65b994a6d91ae2f5ff4fc249d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36165 } attrs { promote: true } }
W20260924 04:14:09.667913 7195 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0d01bc511ef45b399351d4e77c4c85d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } } peers { permanent_uuid: "15bf6683fb84414380dea83f7e7f93d8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45405 } } peers { permanent_uuid: "86c8e5a65b994a6d91ae2f5ff4fc249d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36165 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0d01bc511ef45b399351d4e77c4c85d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } } peers { permanent_uuid: "15bf6683fb84414380dea83f7e7f93d8" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45405 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c0d01bc511ef45b399351d4e77c4c85d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } } peers { permanent_uuid: "15bf6683fb84414380dea83f7e7f93d8" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45405 } attrs { promote: true } } peers { permanent_uuid: "86c8e5a65b994a6d91ae2f5ff4fc249d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36165 } attrs { promote: true } }
I20260924 04:14:09.668903 7113 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 86c8e5a65b994a6d91ae2f5ff4fc249d (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c0d01bc511ef45b399351d4e77c4c85d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } } peers { permanent_uuid: "15bf6683fb84414380dea83f7e7f93d8" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45405 } attrs { promote: true } } peers { permanent_uuid: "86c8e5a65b994a6d91ae2f5ff4fc249d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36165 } attrs { promote: true } } }
I20260924 04:14:09.669376 7195 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 15bf6683fb84414380dea83f7e7f93d8 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0d01bc511ef45b399351d4e77c4c85d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } } peers { permanent_uuid: "15bf6683fb84414380dea83f7e7f93d8" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45405 } attrs { promote: true } } }
I20260924 04:14:09.670747 7142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0d01bc511ef45b399351d4e77c4c85d. Latest consensus state: current_term: 1 leader_uuid: "c0d01bc511ef45b399351d4e77c4c85d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c0d01bc511ef45b399351d4e77c4c85d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } } peers { permanent_uuid: "15bf6683fb84414380dea83f7e7f93d8" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45405 } attrs { promote: true } } peers { permanent_uuid: "86c8e5a65b994a6d91ae2f5ff4fc249d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36165 } attrs { promote: true } } }
I20260924 04:14:09.670872 7142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8 [sys.catalog]: This master's current role is: LEARNER
I20260924 04:14:09.670966 7142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c0d01bc511ef45b399351d4e77c4c85d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c0d01bc511ef45b399351d4e77c4c85d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } } peers { permanent_uuid: "15bf6683fb84414380dea83f7e7f93d8" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45405 } attrs { promote: true } } peers { permanent_uuid: "86c8e5a65b994a6d91ae2f5ff4fc249d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36165 } attrs { promote: true } } }
I20260924 04:14:09.671048 7142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8 [sys.catalog]: This master's current role is: LEARNER
I20260924 04:14:09.671263 7253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c0d01bc511ef45b399351d4e77c4c85d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0d01bc511ef45b399351d4e77c4c85d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } } peers { permanent_uuid: "15bf6683fb84414380dea83f7e7f93d8" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45405 } attrs { promote: true } } }
I20260924 04:14:09.671450 7253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260924 04:14:09.671486 7075 master.cc:537] Master@127.0.98.60:36165 shutting down...
I20260924 04:14:09.671588 7227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0d01bc511ef45b399351d4e77c4c85d. Latest consensus state: current_term: 1 leader_uuid: "c0d01bc511ef45b399351d4e77c4c85d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0d01bc511ef45b399351d4e77c4c85d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } } peers { permanent_uuid: "15bf6683fb84414380dea83f7e7f93d8" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45405 } attrs { promote: true } } }
I20260924 04:14:09.671676 7227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260924 04:14:09.671617 7253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c0d01bc511ef45b399351d4e77c4c85d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0d01bc511ef45b399351d4e77c4c85d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } } peers { permanent_uuid: "15bf6683fb84414380dea83f7e7f93d8" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45405 } attrs { promote: true } } }
I20260924 04:14:09.671919 7253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260924 04:14:09.673079 7061 raft_consensus.cc:1050] T 00000000000000000000000000000000 P c0d01bc511ef45b399351d4e77c4c85d: attempting to promote NON_VOTER 15bf6683fb84414380dea83f7e7f93d8 to VOTER
I20260924 04:14:09.673547 7061 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0d01bc511ef45b399351d4e77c4c85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c0d01bc511ef45b399351d4e77c4c85d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } } peers { permanent_uuid: "15bf6683fb84414380dea83f7e7f93d8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45405 } attrs { promote: false } } peers { permanent_uuid: "86c8e5a65b994a6d91ae2f5ff4fc249d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36165 } attrs { promote: true } }
I20260924 04:14:09.675458 7113 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8 [term 1 LEARNER]: Refusing update from remote peer c0d01bc511ef45b399351d4e77c4c85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260924 04:14:09.675978 7062 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c0d01bc511ef45b399351d4e77c4c85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "15bf6683fb84414380dea83f7e7f93d8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45405 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260924 04:14:09.677063 6490 consensus_peers.cc:487] T 00000000000000000000000000000000 P c0d01bc511ef45b399351d4e77c4c85d -> Peer 86c8e5a65b994a6d91ae2f5ff4fc249d (127.0.98.60:36165): Couldn't send request to peer 86c8e5a65b994a6d91ae2f5ff4fc249d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 04:14:09.677969 7062 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c0d01bc511ef45b399351d4e77c4c85d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 15bf6683fb84414380dea83f7e7f93d8 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c0d01bc511ef45b399351d4e77c4c85d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } } peers { permanent_uuid: "15bf6683fb84414380dea83f7e7f93d8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45405 } attrs { promote: false } } peers { permanent_uuid: "86c8e5a65b994a6d91ae2f5ff4fc249d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36165 } attrs { promote: true } } }
I20260924 04:14:09.678965 7248 mvcc.cc:205] Tried to move back new op lower bound from 7332754425492103168 to 7332754425358237696. Current Snapshot: MvccSnapshot[applied={T|T < 7332754425392865280}]
I20260924 04:14:09.679463 7062 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0d01bc511ef45b399351d4e77c4c85d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c0d01bc511ef45b399351d4e77c4c85d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c0d01bc511ef45b399351d4e77c4c85d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } } peers { permanent_uuid: "15bf6683fb84414380dea83f7e7f93d8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45405 } attrs { promote: false } } peers { permanent_uuid: "86c8e5a65b994a6d91ae2f5ff4fc249d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36165 } attrs { promote: true } } }
I20260924 04:14:09.679577 7062 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0d01bc511ef45b399351d4e77c4c85d [sys.catalog]: This master's current role is: LEADER
I20260924 04:14:09.679731 7062 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0d01bc511ef45b399351d4e77c4c85d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c0d01bc511ef45b399351d4e77c4c85d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c0d01bc511ef45b399351d4e77c4c85d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } } peers { permanent_uuid: "15bf6683fb84414380dea83f7e7f93d8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45405 } attrs { promote: false } } peers { permanent_uuid: "86c8e5a65b994a6d91ae2f5ff4fc249d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36165 } attrs { promote: true } } }
I20260924 04:14:09.679822 7062 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0d01bc511ef45b399351d4e77c4c85d [sys.catalog]: This master's current role is: LEADER
I20260924 04:14:09.680071 7113 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 15bf6683fb84414380dea83f7e7f93d8 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c0d01bc511ef45b399351d4e77c4c85d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } } peers { permanent_uuid: "15bf6683fb84414380dea83f7e7f93d8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45405 } attrs { promote: false } } peers { permanent_uuid: "86c8e5a65b994a6d91ae2f5ff4fc249d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36165 } attrs { promote: true } } }
I20260924 04:14:09.680642 7075 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260924 04:14:09.680868 7075 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260924 04:14:09.680941 7075 tablet_replica.cc:330] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d: stopping tablet replica
I20260924 04:14:09.681582 7252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c0d01bc511ef45b399351d4e77c4c85d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c0d01bc511ef45b399351d4e77c4c85d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } } peers { permanent_uuid: "15bf6683fb84414380dea83f7e7f93d8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45405 } attrs { promote: false } } peers { permanent_uuid: "86c8e5a65b994a6d91ae2f5ff4fc249d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36165 } attrs { promote: true } } }
I20260924 04:14:09.681770 7252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:14:09.703408 7075 master.cc:552] Master@127.0.98.60:36165 shutdown complete.
I20260924 04:14:09.703477 7075 master_runner.cc:305] Clearing existing system tablet
I20260924 04:14:09.703727 7075 fs_manager.cc:359] Metadata directory not provided
I20260924 04:14:09.703786 7075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:14:09.705183 7075 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 04:14:09.706111 7259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:09.706365 7075 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:14:09.706497 7075 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 04:14:09.706542 7075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-2/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-2/wal
uuid: "86c8e5a65b994a6d91ae2f5ff4fc249d"
format_stamp: "Formatted at 2026-09-24 04:14:09 on dist-test-slave-p401"
server_key: "4b32af0b17e35421d72be9272ece1643"
server_key_iv: "4f2ab3397dd384661ae2d2317c45b605"
server_key_version: "encryptionkey@0"
I20260924 04:14:09.708422 7075 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:14:09.710762 7075 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 04:14:09.710855 7075 log.cc:1196] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260924 04:14:09.711149 7075 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d: Deleting consensus metadata
I20260924 04:14:09.711341 7075 master_runner.cc:315] Copying system tablet from 127.0.98.62:34999
I20260924 04:14:09.712517 7075 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:34999
I20260924 04:14:09.721500 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:09.713169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44954 (local address 127.0.98.62:34999)
0924 04:14:09.713305 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:09.713310 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:09.713407 (+ 97us) server_negotiation.cc:407] Connection header received
0924 04:14:09.713552 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:09.713556 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:09.713608 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:09.713715 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:09.714180 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.715027 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.716289 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.716618 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.716728 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:09.718029 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:09.718033 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:09.718036 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:09.718065 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:09.719682 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:09.720175 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:09.720179 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:09.720181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:09.720267 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:09.720558 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:09.720562 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:09.720563 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:09.720800 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:09.720891 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:09.721212 (+ 321us) server_negotiation.cc:299] Negotiation successful
0924 04:14:09.721327 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 04:14:09.721524 7265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:09.713041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34999 (local address 127.0.0.1:44954)
0924 04:14:09.713326 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:09.713340 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:09.713456 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:09.713732 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:09.713735 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:09.713952 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:09.714065 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:09.714072 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.715216 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.715220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:09.716155 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:09.716166 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.716757 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.716762 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:09.716855 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:09.717404 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:09.717422 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:09.717873 (+ 451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:14:09.719792 (+ 1919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:09.719795 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:09.719798 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:09.720057 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:09.720354 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:09.720357 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:09.720359 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:09.720458 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:09.720912 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:14:09.720916 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:14:09.721026 (+ 110us) client_negotiation.cc:769] Sending connection context
0924 04:14:09.721157 (+ 131us) client_negotiation.cc:240] Negotiation successful
0924 04:14:09.721309 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":82,"threads_started":1}
I20260924 04:14:09.722419 6523 tablet_copy_service.cc:140] P c0d01bc511ef45b399351d4e77c4c85d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 86c8e5a65b994a6d91ae2f5ff4fc249d ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:44954)
I20260924 04:14:09.722537 6523 tablet_copy_service.cc:161] P c0d01bc511ef45b399351d4e77c4c85d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 86c8e5a65b994a6d91ae2f5ff4fc249d at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:44954: session id = 86c8e5a65b994a6d91ae2f5ff4fc249d-00000000000000000000000000000000
I20260924 04:14:09.723690 6523 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c0d01bc511ef45b399351d4e77c4c85d: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 04:14:09.724967 7075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 04:14:09.727499 7075 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d: tablet copy: Starting download of 0 data blocks...
I20260924 04:14:09.727706 7075 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d: tablet copy: Starting download of 1 WAL segments...
I20260924 04:14:09.729317 7075 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 04:14:09.731205 6523 tablet_copy_service.cc:342] P c0d01bc511ef45b399351d4e77c4c85d: Request end of tablet copy session 86c8e5a65b994a6d91ae2f5ff4fc249d-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:44954
I20260924 04:14:09.731382 6523 tablet_copy_service.cc:433] P c0d01bc511ef45b399351d4e77c4c85d: ending tablet copy session 86c8e5a65b994a6d91ae2f5ff4fc249d-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 86c8e5a65b994a6d91ae2f5ff4fc249d
I20260924 04:14:09.738581 7075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:14:09.740247 7266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:09.740288 7269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:09.743393 7267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:09.744364 7075 server_base.cc:1034] running on GCE node
I20260924 04:14:09.744577 7075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:14:09.744910 7075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:14:09.746066 7075 hybrid_clock.cc:648] HybridClock initialized: now 1790223249746048 us; error 35 us; skew 500 ppm
Sep 24 04:14:09 dist-test-slave-p401 krb5kdc[6458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223249, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:14:09.747725 7075 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260924 04:14:09.748160 7075 webserver.cc:460] Webserver started at http://127.0.98.60:45281/ using document root <none> and password file <none>
I20260924 04:14:09.748338 7075 fs_manager.cc:359] Metadata directory not provided
I20260924 04:14:09.748381 7075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:14:09.749646 7075 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 04:14:09.750650 7276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:09.750905 7075 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 04:14:09.750973 7075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-2/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-2/wal
uuid: "86c8e5a65b994a6d91ae2f5ff4fc249d"
format_stamp: "Formatted at 2026-09-24 04:14:09 on dist-test-slave-p401"
server_key: "4b32af0b17e35421d72be9272ece1643"
server_key_iv: "4f2ab3397dd384661ae2d2317c45b605"
server_key_version: "encryptionkey@0"
I20260924 04:14:09.751042 7075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260924 04:14:09.770954 7247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:14:09.771349 7247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:14:09.771451 7247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 04:14:09.773074 7075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:14:09.775215 7075 kserver.cc:163] Server-wide thread pool size limit: 3276
W20260924 04:14:09.777257 7247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:14:09.777324 7247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:14:09.777426 7247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:14:09.779942 7075 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:36165
I20260924 04:14:09.779949 7331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:36165 every 8 connection(s)
I20260924 04:14:09.780505 7075 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/master-2/data/info.pb
I20260924 04:14:09.782694 7332 sys_catalog.cc:263] Verifying existing consensus state
I20260924 04:14:09.783795 7332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d: Bootstrap starting.
I20260924 04:14:09.784333 7247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:36843
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=34333
--tserver_master_addrs=127.0.98.62:34999,127.0.98.61:45405,127.0.98.60:36165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:14:09.785429 7247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:14:09.786434 7247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:14:09.790671 7332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:14:09.790984 7332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d: Bootstrap complete.
I20260924 04:14:09.791504 7332 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c0d01bc511ef45b399351d4e77c4c85d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } } peers { permanent_uuid: "15bf6683fb84414380dea83f7e7f93d8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45405 } attrs { promote: false } } peers { permanent_uuid: "86c8e5a65b994a6d91ae2f5ff4fc249d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36165 } attrs { promote: true } }
I20260924 04:14:09.791630 7332 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 86c8e5a65b994a6d91ae2f5ff4fc249d, State: Initialized, Role: LEARNER
I20260924 04:14:09.791742 7332 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c0d01bc511ef45b399351d4e77c4c85d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } } peers { permanent_uuid: "15bf6683fb84414380dea83f7e7f93d8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45405 } attrs { promote: false } } peers { permanent_uuid: "86c8e5a65b994a6d91ae2f5ff4fc249d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36165 } attrs { promote: true } }
I20260924 04:14:09.792030 7340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c0d01bc511ef45b399351d4e77c4c85d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } } peers { permanent_uuid: "15bf6683fb84414380dea83f7e7f93d8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45405 } attrs { promote: false } } peers { permanent_uuid: "86c8e5a65b994a6d91ae2f5ff4fc249d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36165 } attrs { promote: true } } }
I20260924 04:14:09.792138 7340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d [sys.catalog]: This master's current role is: LEARNER
I20260924 04:14:09.792155 7332 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:14:09.794615 7351 catalog_manager.cc:1259] Loaded cluster ID: 4c5e55d2a7c949cf9287fac82ca19300
I20260924 04:14:09.794664 7351 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d: loading cluster ID for follower catalog manager: success
W20260924 04:14:09.796437 7336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:09.796615 7247 server_base.cc:1034] running on GCE node
W20260924 04:14:09.796438 7337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:09.796550 7339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:09.796840 7351 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d: acquiring CA information for follower catalog manager: success
I20260924 04:14:09.797143 7247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:14:09.797766 7247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:14:09.797963 7351 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 04:14:09.798929 7247 hybrid_clock.cc:648] HybridClock initialized: now 1790223249798901 us; error 45 us; skew 500 ppm
Sep 24 04:14:09 dist-test-slave-p401 krb5kdc[6458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223249, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:14:09.801548 7247 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 04:14:09.802390 7247 webserver.cc:460] Webserver started at http://127.0.98.1:34333/ using document root <none> and password file <none>
I20260924 04:14:09.802920 7247 fs_manager.cc:359] Metadata directory not provided
I20260924 04:14:09.802997 7247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:14:09.806885 7247 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:14:09.809317 7358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:09.810527 7247 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 04:14:09.810680 7247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-0/wal
uuid: "34d2513e892f45b781f55ccdeae8ecd7"
format_stamp: "Formatted at 2026-09-24 04:14:08 on dist-test-slave-p401"
server_key: "b58974f3e198af0e933715bbaa45f9f0"
server_key_iv: "377cc1e49ba182e3bb5399b3d81860f3"
server_key_version: "encryptionkey@0"
I20260924 04:14:09.811249 7247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:14:09.826603 7247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:14:09.830133 7247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:14:09.830312 7247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:14:09.830929 7247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:14:09.831818 7247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:14:09.831871 7247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:09.831908 7247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:14:09.831925 7247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:09.840488 7247 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36843
I20260924 04:14:09.840588 7471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36843 every 8 connection(s)
I20260924 04:14:09.841425 7247 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:14:09.844175 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 7247
I20260924 04:14:09.844373 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 6699
I20260924 04:14:09.851681 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:33567
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34999
--webserver_port=40225
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:34999,127.0.98.61:45405
--tserver_master_addrs=127.0.98.62:34999,127.0.98.61:45405,127.0.98.60:36165 with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 04:14:09 dist-test-slave-p401 krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223249, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 24 04:14:09 dist-test-slave-p401 krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223249, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 24 04:14:09 dist-test-slave-p401 krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223249, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 04:14:09.858037 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:09.843129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49735 (local address 127.0.98.62:34999)
0924 04:14:09.843330 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:09.843333 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:09.844598 (+ 1265us) server_negotiation.cc:407] Connection header received
0924 04:14:09.845760 (+ 1162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:09.845763 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:09.845815 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:09.845907 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:09.847399 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.848235 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.849945 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.850286 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.850360 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:09.854230 (+ 3870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:09.854234 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:09.854237 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:09.854267 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:09.855762 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:09.856346 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:09.856350 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:09.856351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:09.856432 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:09.856811 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:09.856815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:09.856816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:09.857030 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:09.857102 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:09.857763 (+ 661us) server_negotiation.cc:299] Negotiation successful
0924 04:14:09.857882 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260924 04:14:09.858214 7479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:09.845226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33609 (local address 127.0.98.60:36165)
0924 04:14:09.847479 (+ 2253us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:09.847484 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:09.847497 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:14:09.847537 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:09.847540 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:09.847583 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:09.847659 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:09.848149 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.849138 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.850135 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.850460 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.850541 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:09.853833 (+ 3292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:09.853837 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:09.853839 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:09.853869 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:09.856509 (+ 2640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:09.856921 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:09.856924 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:09.856926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:09.856999 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:09.857269 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:09.857272 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:09.857273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:09.857464 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:09.857547 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:09.857960 (+ 413us) server_negotiation.cc:299] Negotiation successful
0924 04:14:09.858070 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2179,"thread_start_us":81,"threads_started":1}
I20260924 04:14:09.859117 7476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:09.843530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34999 (local address 127.0.98.1:49735)
0924 04:14:09.844442 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:09.844497 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:09.845522 (+ 1025us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:09.846102 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:09.846112 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:09.846581 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:09.847217 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:09.847259 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.848398 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.848401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:09.849823 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:09.849833 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.850511 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.850514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:09.850730 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:09.851639 (+ 909us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:09.851653 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:09.854056 (+ 2403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:14:09.855871 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:09.855878 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:09.855893 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:09.856227 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:09.856531 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:09.856534 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:09.856536 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:09.856702 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:09.857157 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:14:09.857163 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:14:09.857488 (+ 325us) client_negotiation.cc:769] Sending connection context
0924 04:14:09.858669 (+ 1181us) client_negotiation.cc:240] Negotiation successful
0924 04:14:09.858793 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"mutex_wait_us":88,"spinlock_wait_cycles":4352,"thread_start_us":512,"threads_started":1}
I20260924 04:14:09.859426 7477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:09.844839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45405 (local address 127.0.98.1:41507)
0924 04:14:09.846508 (+ 1669us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:09.846519 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:09.846583 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:09.847395 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:09.847398 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:09.847585 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:09.847705 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:09.847711 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.848699 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.848702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:09.849824 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:09.849841 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.850494 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.850498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:09.850738 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:09.851475 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:09.851496 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:09.854456 (+ 2960us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:14:09.856253 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:09.856256 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:09.856259 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:09.856508 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:09.856803 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:09.856806 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:09.856808 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:09.856890 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:09.857329 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:14:09.857344 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:14:09.857490 (+ 146us) client_negotiation.cc:769] Sending connection context
0924 04:14:09.857700 (+ 210us) client_negotiation.cc:240] Negotiation successful
0924 04:14:09.858052 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1616,"mutex_wait_us":96,"spinlock_wait_cycles":3200,"thread_start_us":67,"threads_started":1}
I20260924 04:14:09.859128 7478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:09.845112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36165 (local address 127.0.98.1:33609)
0924 04:14:09.846704 (+ 1592us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:09.846712 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:09.846764 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:09.847763 (+ 999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:09.847766 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:09.847925 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:09.848024 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:09.848030 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.849249 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.849253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:09.850020 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:09.850027 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.850656 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.850660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:09.850755 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:09.851531 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:09.851546 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:09.853659 (+ 2113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:14:09.856608 (+ 2949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:09.856610 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:09.856612 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:09.856828 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:09.857092 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:09.857094 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:09.857096 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:09.857176 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:09.857587 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:14:09.857590 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:14:09.857691 (+ 101us) client_negotiation.cc:769] Sending connection context
0924 04:14:09.857785 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 04:14:09.858445 (+ 660us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1545,"mutex_wait_us":656,"thread_start_us":69,"threads_started":1}
I20260924 04:14:09.859966 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:09.846971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41507 (local address 127.0.98.61:45405)
0924 04:14:09.847089 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:09.847093 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:09.847106 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:14:09.847144 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:09.847147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:09.847191 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:09.847286 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:09.847825 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.848579 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.849950 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:09.850250 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:09.850312 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:09.854592 (+ 4280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:09.854595 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:09.854598 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:09.854623 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:09.856131 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:09.856618 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:09.856621 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:09.856623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:09.856706 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:09.856989 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:09.856993 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:09.856994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:09.857199 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:09.857287 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:09.859708 (+ 2421us) server_negotiation.cc:299] Negotiation successful
0924 04:14:09.859822 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 04:14:09.860646 7472 heartbeater.cc:347] Connected to a master server at 127.0.98.60:36165
I20260924 04:14:09.860882 7472 heartbeater.cc:464] Registering TS with master...
I20260924 04:14:09.860960 7474 heartbeater.cc:347] Connected to a master server at 127.0.98.61:45405
I20260924 04:14:09.861042 7474 heartbeater.cc:464] Registering TS with master...
I20260924 04:14:09.861413 7472 heartbeater.cc:511] Master 127.0.98.60:36165 requested a full tablet report, sending...
I20260924 04:14:09.861411 7474 heartbeater.cc:511] Master 127.0.98.61:45405 requested a full tablet report, sending...
I20260924 04:14:09.861505 7473 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34999
I20260924 04:14:09.861560 7473 heartbeater.cc:464] Registering TS with master...
I20260924 04:14:09.861697 7473 heartbeater.cc:511] Master 127.0.98.62:34999 requested a full tablet report, sending...
I20260924 04:14:09.862762 6503 ts_manager.cc:194] Re-registered known tserver with Master: 34d2513e892f45b781f55ccdeae8ecd7 (127.0.98.1:36843)
I20260924 04:14:09.862933 7103 ts_manager.cc:194] Registered new tserver with Master: 34d2513e892f45b781f55ccdeae8ecd7 (127.0.98.1:36843)
I20260924 04:14:09.863533 6503 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49735
I20260924 04:14:09.863867 7296 ts_manager.cc:194] Registered new tserver with Master: 34d2513e892f45b781f55ccdeae8ecd7 (127.0.98.1:36843)
W20260924 04:14:09.961843 7480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:14:09.962122 7480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:14:09.962184 7480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:14:09.966264 7480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:14:09.966336 7480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:14:09.966415 7480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:14:09.971721 7480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:33567
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=40225
--tserver_master_addrs=127.0.98.62:34999,127.0.98.61:45405,127.0.98.60:36165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:14:09.972761 7480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:14:09.973768 7480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:14:09.983213 7485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:09.983354 7480 server_base.cc:1034] running on GCE node
W20260924 04:14:09.983223 7488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:09.983213 7486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:09.983886 7480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:14:09.984477 7480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:14:09.985630 7480 hybrid_clock.cc:648] HybridClock initialized: now 1790223249985604 us; error 36 us; skew 500 ppm
Sep 24 04:14:09 dist-test-slave-p401 krb5kdc[6458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223249, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:14:09.988153 7480 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 04:14:09.988956 7480 webserver.cc:460] Webserver started at http://127.0.98.2:40225/ using document root <none> and password file <none>
I20260924 04:14:09.989475 7480 fs_manager.cc:359] Metadata directory not provided
I20260924 04:14:09.989552 7480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:14:09.993366 7480 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:14:09.995378 7495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:09.996532 7480 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:14:09.996634 7480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-1/wal
uuid: "12f50fe1de024fe1b72357e102c83015"
format_stamp: "Formatted at 2026-09-24 04:14:08 on dist-test-slave-p401"
server_key: "3dff6e2b3ca7eac3fe48e6665007753a"
server_key_iv: "f0d5455bca9f26941ec41bc6de4f8470"
server_key_version: "encryptionkey@0"
I20260924 04:14:09.997136 7480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:14:10.018307 7480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:14:10.021267 7480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:14:10.021445 7480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:14:10.022037 7480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:14:10.022873 7480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:14:10.022917 7480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:10.022971 7480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:14:10.023000 7480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:10.032401 7480 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33567
I20260924 04:14:10.032439 7608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33567 every 8 connection(s)
I20260924 04:14:10.033411 7480 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:14:10.038241 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 7480
I20260924 04:14:10.038435 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 6833
Sep 24 04:14:10 dist-test-slave-p401 krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223249, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 24 04:14:10 dist-test-slave-p401 krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223249, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 24 04:14:10 dist-test-slave-p401 krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223249, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 04:14:10.047413 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41945
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34999
--webserver_port=35857
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:34999,127.0.98.61:45405
--tserver_master_addrs=127.0.98.62:34999,127.0.98.61:45405,127.0.98.60:36165 with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:14:10.049845 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:10.037210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42913 (local address 127.0.98.62:34999)
0924 04:14:10.037319 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:10.037323 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:10.037341 (+ 18us) server_negotiation.cc:407] Connection header received
0924 04:14:10.037879 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:10.037882 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:10.037954 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:10.038024 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:10.039340 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.040084 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.041984 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.042313 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.042383 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:10.045605 (+ 3222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:10.045608 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:10.045611 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:10.045639 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:10.047897 (+ 2258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:10.048318 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:10.048321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:10.048323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:10.048400 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:10.048678 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:10.048681 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:10.048683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:10.048899 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:10.048967 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:10.049550 (+ 583us) server_negotiation.cc:299] Negotiation successful
0924 04:14:10.049669 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 04:14:10.049885 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:10.036854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46191 (local address 127.0.98.61:45405)
0924 04:14:10.036972 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:10.036977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:10.037080 (+ 103us) server_negotiation.cc:407] Connection header received
0924 04:14:10.037572 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:10.037575 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:10.037627 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:10.037725 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:10.039308 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.040122 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.041970 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.042304 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.042376 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:10.045763 (+ 3387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:10.045767 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:10.045770 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:10.045798 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:10.047477 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:10.048054 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:10.048058 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:10.048059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:10.048146 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:10.048593 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:10.048596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:10.048598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:10.048824 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:10.048890 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:10.049623 (+ 733us) server_negotiation.cc:299] Negotiation successful
0924 04:14:10.049735 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 04:14:10.049912 7479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:10.035089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46559 (local address 127.0.98.60:36165)
0924 04:14:10.035213 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:10.035217 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:10.036450 (+ 1233us) server_negotiation.cc:407] Connection header received
0924 04:14:10.037686 (+ 1236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:10.037689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:10.037760 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:10.037838 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:10.039473 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.040213 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.041969 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.042311 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.042392 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:10.046741 (+ 4349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:10.046744 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:10.046747 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:10.046778 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:10.048326 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:10.048730 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:10.048733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:10.048735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:10.048807 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:10.049089 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:10.049092 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:10.049094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:10.049289 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:10.049367 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:10.049661 (+ 294us) server_negotiation.cc:299] Negotiation successful
0924 04:14:10.049768 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 04:14:10.051577 7613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:10.035495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36165 (local address 127.0.98.2:46559)
0924 04:14:10.036308 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:10.036348 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:10.037375 (+ 1027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:10.037894 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:10.037904 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:10.038564 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:10.039359 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:10.039366 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.040330 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.040345 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:10.041836 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:10.041847 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.042565 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.042569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:10.042847 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:10.043735 (+ 888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:10.043747 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:10.046565 (+ 2818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:14:10.048430 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:10.048433 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:10.048435 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:10.048636 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:10.048919 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:10.048922 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:10.048923 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:10.048999 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:10.049418 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:14:10.049421 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:14:10.049492 (+ 71us) client_negotiation.cc:769] Sending connection context
0924 04:14:10.050553 (+ 1061us) client_negotiation.cc:240] Negotiation successful
0924 04:14:10.050660 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"mutex_wait_us":291,"spinlock_wait_cycles":36608,"thread_start_us":498,"threads_started":1}
I20260924 04:14:10.051587 7614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:10.036752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45405 (local address 127.0.98.2:46191)
0924 04:14:10.037002 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:10.037015 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:10.037380 (+ 365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:10.037894 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:10.037904 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:10.038505 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:10.039160 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:10.039175 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.040253 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.040256 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:10.041851 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:10.041858 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.042565 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.042569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:10.042848 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:10.043638 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:10.043650 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:10.045633 (+ 1983us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:14:10.047600 (+ 1967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:10.047606 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:10.047623 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:10.047941 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:10.048253 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:10.048256 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:10.048258 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:10.048491 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:10.048960 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:14:10.048966 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:14:10.049283 (+ 317us) client_negotiation.cc:769] Sending connection context
0924 04:14:10.050085 (+ 802us) client_negotiation.cc:240] Negotiation successful
0924 04:14:10.050559 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":452,"spinlock_wait_cycles":31232,"thread_start_us":66,"threads_started":1}
I20260924 04:14:10.051890 7615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:10.037078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34999 (local address 127.0.98.2:42913)
0924 04:14:10.037282 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:10.037294 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:10.037379 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:10.038145 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:10.038147 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:10.038497 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:10.039162 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:10.039176 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.040250 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.040253 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:10.041859 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:10.041870 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.042565 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.042568 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:10.042840 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:10.043620 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:10.043641 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:10.045422 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:14:10.048005 (+ 2583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:10.048008 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:10.048009 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:10.048220 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:10.048494 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:10.048497 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:10.048498 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:10.048582 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:10.049032 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:14:10.049035 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:14:10.049282 (+ 247us) client_negotiation.cc:769] Sending connection context
0924 04:14:10.050098 (+ 816us) client_negotiation.cc:240] Negotiation successful
0924 04:14:10.050485 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":337,"spinlock_wait_cycles":6144,"thread_start_us":88,"threads_started":1}
I20260924 04:14:10.053078 7611 heartbeater.cc:347] Connected to a master server at 127.0.98.61:45405
I20260924 04:14:10.053171 7609 heartbeater.cc:347] Connected to a master server at 127.0.98.60:36165
I20260924 04:14:10.053179 7610 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34999
I20260924 04:14:10.053308 7610 heartbeater.cc:464] Registering TS with master...
I20260924 04:14:10.053308 7609 heartbeater.cc:464] Registering TS with master...
I20260924 04:14:10.053884 7609 heartbeater.cc:511] Master 127.0.98.60:36165 requested a full tablet report, sending...
I20260924 04:14:10.053884 7610 heartbeater.cc:511] Master 127.0.98.62:34999 requested a full tablet report, sending...
I20260924 04:14:10.053308 7611 heartbeater.cc:464] Registering TS with master...
I20260924 04:14:10.054539 7611 heartbeater.cc:511] Master 127.0.98.61:45405 requested a full tablet report, sending...
I20260924 04:14:10.055078 6503 ts_manager.cc:194] Re-registered known tserver with Master: 12f50fe1de024fe1b72357e102c83015 (127.0.98.2:33567)
I20260924 04:14:10.055138 7103 ts_manager.cc:194] Registered new tserver with Master: 12f50fe1de024fe1b72357e102c83015 (127.0.98.2:33567)
I20260924 04:14:10.055274 7296 ts_manager.cc:194] Registered new tserver with Master: 12f50fe1de024fe1b72357e102c83015 (127.0.98.2:33567)
I20260924 04:14:10.055804 6503 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42913
W20260924 04:14:10.157725 7616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:14:10.157975 7616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:14:10.158034 7616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:14:10.161969 7616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:14:10.162025 7616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:14:10.162106 7616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:14:10.167407 7616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41945
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=35857
--tserver_master_addrs=127.0.98.62:34999,127.0.98.61:45405,127.0.98.60:36165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:14:10.168448 7616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:14:10.169461 7616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:14:10.178798 7621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:10.178824 7624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:10.178856 7622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:10.178958 7616 server_base.cc:1034] running on GCE node
I20260924 04:14:10.179467 7616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:14:10.180193 7616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:14:10.181344 7616 hybrid_clock.cc:648] HybridClock initialized: now 1790223250181322 us; error 38 us; skew 500 ppm
Sep 24 04:14:10 dist-test-slave-p401 krb5kdc[6458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223250, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:14:10.183866 7616 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 04:14:10.184667 7616 webserver.cc:460] Webserver started at http://127.0.98.3:35857/ using document root <none> and password file <none>
I20260924 04:14:10.185176 7616 fs_manager.cc:359] Metadata directory not provided
I20260924 04:14:10.185251 7616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:14:10.188882 7616 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:14:10.190722 7631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:10.191969 7616 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:14:10.192067 7616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-2/wal
uuid: "7627c004b34c46e89a393afbfb63f656"
format_stamp: "Formatted at 2026-09-24 04:14:08 on dist-test-slave-p401"
server_key: "a73219c680d373174eb66acb01e4f96f"
server_key_iv: "80a10390833332016f00f4d794c50ebe"
server_key_version: "encryptionkey@0"
I20260924 04:14:10.192565 7616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:14:10.214762 7616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:14:10.217974 7616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:14:10.218152 7616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:14:10.218748 7616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:14:10.219657 7616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:14:10.219703 7616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:10.219756 7616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:14:10.219790 7616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:10.228140 7616 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41945
I20260924 04:14:10.228163 7744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41945 every 8 connection(s)
I20260924 04:14:10.229199 7616 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:14:10.233889 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 7616
Sep 24 04:14:10 dist-test-slave-p401 krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223250, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 24 04:14:10 dist-test-slave-p401 krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223250, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 24 04:14:10 dist-test-slave-p401 krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223250, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 04:14:10.245174 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:10.236380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54830 (local address 127.0.98.61:45405)
0924 04:14:10.236716 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:10.236720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:10.236731 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:14:10.236766 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:10.236769 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:10.236809 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:10.236888 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:10.237845 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.238603 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.239709 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.240014 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.240112 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:10.241558 (+ 1446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:10.241562 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:10.241564 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:10.241593 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:10.243396 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:10.243855 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:10.243858 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:10.243860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:10.243944 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:10.244229 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:10.244234 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:10.244236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:10.244451 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:10.244815 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:10.244888 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 04:14:10.245000 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":73,"threads_started":1}
I20260924 04:14:10.248202 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:10.232610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46575 (local address 127.0.98.62:34999)
0924 04:14:10.232722 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:10.232726 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:10.232899 (+ 173us) server_negotiation.cc:407] Connection header received
0924 04:14:10.233846 (+ 947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:10.233849 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:10.233896 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:10.233957 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:10.235727 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.236429 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.239363 (+ 2934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.239682 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.239765 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:10.243271 (+ 3506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:10.243275 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:10.243278 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:10.243305 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:10.246537 (+ 3232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:10.246941 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:10.246944 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:10.246946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:10.247028 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:10.247323 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:10.247326 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:10.247328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:10.247579 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:10.247672 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:10.247949 (+ 277us) server_negotiation.cc:299] Negotiation successful
0924 04:14:10.248056 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":45}
I20260924 04:14:10.248320 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:10.232935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44457 (local address 127.0.98.61:45405)
0924 04:14:10.233051 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:10.233055 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:10.233340 (+ 285us) server_negotiation.cc:407] Connection header received
0924 04:14:10.233766 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:10.233770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:10.233823 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:10.233875 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:10.237434 (+ 3559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.238191 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.241082 (+ 2891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.241385 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.241481 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:10.244032 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:10.244036 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:10.244038 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:10.244060 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:10.245538 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:10.246029 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:10.246032 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:10.246034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:10.246102 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:10.246447 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:10.246450 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:10.246452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:10.246609 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:10.246691 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:10.248098 (+ 1407us) server_negotiation.cc:299] Negotiation successful
0924 04:14:10.248196 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:14:10.248816 7752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:10.232850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45405 (local address 127.0.98.3:44457)
0924 04:14:10.233274 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:10.233287 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:10.233554 (+ 267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:10.234215 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:10.234224 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:10.234884 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:10.235590 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:10.235611 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.240172 (+ 4561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.240175 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:10.240950 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:10.240959 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.242151 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.242155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:10.242236 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:10.242645 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:10.242658 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:10.243933 (+ 1275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:14:10.245648 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:10.245654 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:10.245672 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:10.245927 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:10.246177 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:10.246180 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:10.246182 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:10.246317 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:10.246739 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:14:10.246746 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:14:10.247073 (+ 327us) client_negotiation.cc:769] Sending connection context
0924 04:14:10.247323 (+ 250us) client_negotiation.cc:240] Negotiation successful
0924 04:14:10.247673 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":46,"spinlock_wait_cycles":7680,"thread_start_us":302,"threads_started":1}
I20260924 04:14:10.250000 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:10.231265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36165 (local address 127.0.98.3:50017)
0924 04:14:10.232105 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:10.232145 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:10.233544 (+ 1399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:10.234214 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:10.234223 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:10.234809 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:10.235600 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:10.235611 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.237436 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.237439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:10.238864 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:10.238874 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.239572 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.239576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:10.239765 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:10.240410 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:10.240431 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:10.242167 (+ 1736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:14:10.248533 (+ 6366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:10.248536 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:10.248538 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:10.248751 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:10.249039 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:10.249042 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:10.249044 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:10.249120 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:10.249560 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:14:10.249563 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:14:10.249647 (+ 84us) client_negotiation.cc:769] Sending connection context
0924 04:14:10.249727 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 04:14:10.249847 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"mutex_wait_us":86,"spinlock_wait_cycles":3200,"thread_start_us":508,"threads_started":1}
I20260924 04:14:10.250311 7748 heartbeater.cc:347] Connected to a master server at 127.0.98.61:45405
I20260924 04:14:10.250506 7479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:10.230877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50017 (local address 127.0.98.60:36165)
0924 04:14:10.230984 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:10.230987 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:10.232236 (+ 1249us) server_negotiation.cc:407] Connection header received
0924 04:14:10.233767 (+ 1531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:10.233771 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:10.233827 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:10.233893 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:10.236561 (+ 2668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.237295 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.239000 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.239325 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.239414 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:10.243606 (+ 4192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:10.243610 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:10.243612 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:10.243637 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:10.248433 (+ 4796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:10.248862 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:10.248865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:10.248867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:10.248944 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:10.249212 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:10.249215 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:10.249217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:10.249436 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:10.249520 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:10.250219 (+ 699us) server_negotiation.cc:299] Negotiation successful
0924 04:14:10.250354 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 04:14:10.250550 7748 heartbeater.cc:464] Registering TS with master...
I20260924 04:14:10.251107 7748 heartbeater.cc:511] Master 127.0.98.61:45405 requested a full tablet report, sending...
I20260924 04:14:10.251168 7746 heartbeater.cc:347] Connected to a master server at 127.0.98.60:36165
I20260924 04:14:10.251253 7746 heartbeater.cc:464] Registering TS with master...
I20260924 04:14:10.251418 7746 heartbeater.cc:511] Master 127.0.98.60:36165 requested a full tablet report, sending...
I20260924 04:14:10.252012 7761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:10.236715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41680 (local address 127.0.98.60:36165)
0924 04:14:10.236903 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:10.236907 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:10.237161 (+ 254us) server_negotiation.cc:407] Connection header received
0924 04:14:10.237299 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:10.237302 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:10.237346 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:10.237419 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:10.238252 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.238975 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.240663 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.240994 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.241071 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:10.242683 (+ 1612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:10.242687 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:10.242690 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:10.242717 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:10.247814 (+ 5097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:10.250592 (+ 2778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:10.250597 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:10.250599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:10.250686 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:10.251027 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:10.251030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:10.251032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:10.251252 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:10.251365 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:10.251724 (+ 359us) server_negotiation.cc:299] Negotiation successful
0924 04:14:10.251836 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":63,"threads_started":1}
I20260924 04:14:10.248819 7751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:10.232497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34999 (local address 127.0.98.3:46575)
0924 04:14:10.232773 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:10.232792 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:10.233544 (+ 752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:10.234214 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:10.234223 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:10.234809 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:10.235625 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:10.235632 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.238334 (+ 2702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.238338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:10.239209 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:10.239218 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.239882 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.239886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:10.239971 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:10.241620 (+ 1649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:10.241634 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:10.243100 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:14:10.246634 (+ 3534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:10.246637 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:10.246639 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:10.246841 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:10.247121 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:10.247124 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:10.247125 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:10.247199 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:10.247666 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:14:10.247670 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:14:10.247761 (+ 91us) client_negotiation.cc:769] Sending connection context
0924 04:14:10.247869 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 04:14:10.247996 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":307,"thread_start_us":83,"threads_started":1}
I20260924 04:14:10.252218 7103 ts_manager.cc:194] Registered new tserver with Master: 7627c004b34c46e89a393afbfb63f656 (127.0.98.3:41945)
I20260924 04:14:10.252745 7296 ts_manager.cc:194] Registered new tserver with Master: 7627c004b34c46e89a393afbfb63f656 (127.0.98.3:41945)
I20260924 04:14:10.252887 7747 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34999
I20260924 04:14:10.252964 7747 heartbeater.cc:464] Registering TS with master...
I20260924 04:14:10.253105 7747 heartbeater.cc:511] Master 127.0.98.62:34999 requested a full tablet report, sending...
I20260924 04:14:10.253665 6503 ts_manager.cc:194] Re-registered known tserver with Master: 7627c004b34c46e89a393afbfb63f656 (127.0.98.3:41945)
I20260924 04:14:10.253937 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:10.235979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44964 (local address 127.0.98.62:34999)
0924 04:14:10.242518 (+ 6539us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:10.242555 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:10.242569 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:14:10.244033 (+ 1464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:10.244036 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:10.244089 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:10.244813 (+ 724us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:10.244919 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.245563 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.246668 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.247021 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.247506 (+ 485us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:10.249455 (+ 1949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:10.249459 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:10.249462 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:10.249486 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:10.251742 (+ 2256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:10.252485 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:10.252488 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:10.252490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:10.252572 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:10.252895 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:10.252898 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:10.252900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:10.253097 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:10.253208 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:10.253684 (+ 476us) server_negotiation.cc:299] Negotiation successful
0924 04:14:10.253798 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":6461,"thread_start_us":80,"threads_started":1}
I20260924 04:14:10.254276 6503 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46575
I20260924 04:14:10.256505 6502 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44964:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:14:10.259460 6502 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:14:10.262646 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:10.259145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36165 (local address 127.0.0.1:41690)
0924 04:14:10.259398 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:10.259412 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:10.259508 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:10.259816 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:10.259820 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:10.259839 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:14:10.259947 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:10.259953 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.260659 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.260662 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:10.261543 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:10.261554 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.262249 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.262253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:10.262349 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:10.262375 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 04:14:10.262451 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 04:14:10.262519 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":122,"threads_started":1}
I20260924 04:14:10.262702 7761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:10.259272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41690 (local address 127.0.98.60:36165)
0924 04:14:10.259397 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:10.259401 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:10.259499 (+ 98us) server_negotiation.cc:407] Connection header received
0924 04:14:10.259633 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:10.259635 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:10.259698 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:10.259784 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:14:10.260068 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.260562 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.261658 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.262142 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.262225 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:10.262332 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:10.262499 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 04:14:10.262577 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 04:14:10.263885 7297 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260924 04:14:10.264245 7340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0d01bc511ef45b399351d4e77c4c85d. Latest consensus state: current_term: 1 leader_uuid: "c0d01bc511ef45b399351d4e77c4c85d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c0d01bc511ef45b399351d4e77c4c85d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } } peers { permanent_uuid: "15bf6683fb84414380dea83f7e7f93d8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45405 } attrs { promote: false } } peers { permanent_uuid: "86c8e5a65b994a6d91ae2f5ff4fc249d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36165 } attrs { promote: true } } }
I20260924 04:14:10.264374 7340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d [sys.catalog]: This master's current role is: LEARNER
I20260924 04:14:10.264794 7766 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0d01bc511ef45b399351d4e77c4c85d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c0d01bc511ef45b399351d4e77c4c85d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c0d01bc511ef45b399351d4e77c4c85d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } } peers { permanent_uuid: "15bf6683fb84414380dea83f7e7f93d8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45405 } attrs { promote: false } } peers { permanent_uuid: "86c8e5a65b994a6d91ae2f5ff4fc249d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36165 } attrs { promote: true } } }
I20260924 04:14:10.264930 7766 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0d01bc511ef45b399351d4e77c4c85d [sys.catalog]: This master's current role is: LEADER
I20260924 04:14:10.266510 7766 raft_consensus.cc:1050] T 00000000000000000000000000000000 P c0d01bc511ef45b399351d4e77c4c85d: attempting to promote NON_VOTER 86c8e5a65b994a6d91ae2f5ff4fc249d to VOTER
I20260924 04:14:10.266831 7766 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0d01bc511ef45b399351d4e77c4c85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c0d01bc511ef45b399351d4e77c4c85d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } } peers { permanent_uuid: "15bf6683fb84414380dea83f7e7f93d8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45405 } attrs { promote: false } } peers { permanent_uuid: "86c8e5a65b994a6d91ae2f5ff4fc249d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36165 } attrs { promote: false } }
I20260924 04:14:10.267904 7113 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8 [term 1 FOLLOWER]: Refusing update from remote peer c0d01bc511ef45b399351d4e77c4c85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260924 04:14:10.268074 7297 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d [term 1 LEARNER]: Refusing update from remote peer c0d01bc511ef45b399351d4e77c4c85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260924 04:14:10.268241 7767 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c0d01bc511ef45b399351d4e77c4c85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "15bf6683fb84414380dea83f7e7f93d8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45405 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260924 04:14:10.268654 7767 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c0d01bc511ef45b399351d4e77c4c85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "86c8e5a65b994a6d91ae2f5ff4fc249d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36165 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260924 04:14:10.269622 7766 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c0d01bc511ef45b399351d4e77c4c85d [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 86c8e5a65b994a6d91ae2f5ff4fc249d (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c0d01bc511ef45b399351d4e77c4c85d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } } peers { permanent_uuid: "15bf6683fb84414380dea83f7e7f93d8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45405 } attrs { promote: false } } peers { permanent_uuid: "86c8e5a65b994a6d91ae2f5ff4fc249d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36165 } attrs { promote: false } } }
I20260924 04:14:10.270936 7767 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0d01bc511ef45b399351d4e77c4c85d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c0d01bc511ef45b399351d4e77c4c85d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c0d01bc511ef45b399351d4e77c4c85d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } } peers { permanent_uuid: "15bf6683fb84414380dea83f7e7f93d8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45405 } attrs { promote: false } } peers { permanent_uuid: "86c8e5a65b994a6d91ae2f5ff4fc249d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36165 } attrs { promote: false } } }
I20260924 04:14:10.271040 7767 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0d01bc511ef45b399351d4e77c4c85d [sys.catalog]: This master's current role is: LEADER
I20260924 04:14:10.271266 7766 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0d01bc511ef45b399351d4e77c4c85d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c0d01bc511ef45b399351d4e77c4c85d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c0d01bc511ef45b399351d4e77c4c85d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } } peers { permanent_uuid: "15bf6683fb84414380dea83f7e7f93d8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45405 } attrs { promote: false } } peers { permanent_uuid: "86c8e5a65b994a6d91ae2f5ff4fc249d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36165 } attrs { promote: false } } }
I20260924 04:14:10.271389 7766 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0d01bc511ef45b399351d4e77c4c85d [sys.catalog]: This master's current role is: LEADER
I20260924 04:14:10.271349 7113 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 86c8e5a65b994a6d91ae2f5ff4fc249d (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c0d01bc511ef45b399351d4e77c4c85d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } } peers { permanent_uuid: "15bf6683fb84414380dea83f7e7f93d8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45405 } attrs { promote: false } } peers { permanent_uuid: "86c8e5a65b994a6d91ae2f5ff4fc249d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36165 } attrs { promote: false } } }
I20260924 04:14:10.271559 7745 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0d01bc511ef45b399351d4e77c4c85d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c0d01bc511ef45b399351d4e77c4c85d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c0d01bc511ef45b399351d4e77c4c85d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } } peers { permanent_uuid: "15bf6683fb84414380dea83f7e7f93d8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45405 } attrs { promote: false } } peers { permanent_uuid: "86c8e5a65b994a6d91ae2f5ff4fc249d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36165 } attrs { promote: false } } }
I20260924 04:14:10.271677 7745 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0d01bc511ef45b399351d4e77c4c85d [sys.catalog]: This master's current role is: LEADER
I20260924 04:14:10.272154 7297 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 86c8e5a65b994a6d91ae2f5ff4fc249d (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c0d01bc511ef45b399351d4e77c4c85d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } } peers { permanent_uuid: "15bf6683fb84414380dea83f7e7f93d8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45405 } attrs { promote: false } } peers { permanent_uuid: "86c8e5a65b994a6d91ae2f5ff4fc249d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36165 } attrs { promote: false } } }
I20260924 04:14:10.273082 7774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c0d01bc511ef45b399351d4e77c4c85d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c0d01bc511ef45b399351d4e77c4c85d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } } peers { permanent_uuid: "15bf6683fb84414380dea83f7e7f93d8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45405 } attrs { promote: false } } peers { permanent_uuid: "86c8e5a65b994a6d91ae2f5ff4fc249d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36165 } attrs { promote: false } } }
I20260924 04:14:10.273376 7774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:14:10.273602 7340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c0d01bc511ef45b399351d4e77c4c85d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c0d01bc511ef45b399351d4e77c4c85d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } } peers { permanent_uuid: "15bf6683fb84414380dea83f7e7f93d8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45405 } attrs { promote: false } } peers { permanent_uuid: "86c8e5a65b994a6d91ae2f5ff4fc249d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36165 } attrs { promote: false } } }
I20260924 04:14:10.273703 7340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:14:10.279855 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:10.273932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41945 (local address 127.0.0.1:49172)
0924 04:14:10.275011 (+ 1079us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:10.275025 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:10.275105 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:10.276355 (+ 1250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:10.276358 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:10.276374 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:14:10.276449 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:10.276454 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.277862 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.277865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:10.278604 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:10.278613 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.279513 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.279517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:10.279601 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:10.279628 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 04:14:10.279669 (+ 41us) client_negotiation.cc:240] Negotiation successful
0924 04:14:10.279732 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1009}
I20260924 04:14:10.280326 7777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:10.274155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36843 (local address 127.0.0.1:58116)
0924 04:14:10.275374 (+ 1219us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:10.275389 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:10.275480 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:10.276203 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:10.276206 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:10.276225 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:14:10.276330 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:10.276352 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.277733 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.277735 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:10.278680 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:10.278688 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.279830 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.279843 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:10.280081 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:10.280100 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 04:14:10.280150 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 04:14:10.280215 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1155,"mutex_wait_us":1,"thread_start_us":71,"threads_started":1}
I20260924 04:14:10.280608 7775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:10.274104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49172 (local address 127.0.98.3:41945)
0924 04:14:10.274448 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:10.274464 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:10.275946 (+ 1482us) server_negotiation.cc:407] Connection header received
0924 04:14:10.276032 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:10.276036 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:10.276207 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:10.276589 (+ 382us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:14:10.276707 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.277752 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.278719 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.279390 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.280095 (+ 705us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:10.280230 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:10.280290 (+ 60us) server_negotiation.cc:299] Negotiation successful
0924 04:14:10.280377 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":101,"threads_started":1}
I20260924 04:14:10.280773 7779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:10.275417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58116 (local address 127.0.98.1:36843)
0924 04:14:10.275703 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:10.275714 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:10.275736 (+ 22us) server_negotiation.cc:407] Connection header received
0924 04:14:10.275827 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:10.275833 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:10.276077 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:10.276235 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:14:10.276434 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.277620 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.278795 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.279540 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.279692 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:10.280526 (+ 834us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:10.280574 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 04:14:10.280641 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":107,"threads_started":1}
I20260924 04:14:10.283578 7406 tablet_service.cc:1467] Processing CreateTablet for tablet 862e4fb4e4d04b6a8a229f6cf76a14b3 (DEFAULT_TABLE table=test-table [id=4183f02bc39c45ad8bcb0842b73d5e1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:14:10.283581 7679 tablet_service.cc:1467] Processing CreateTablet for tablet 862e4fb4e4d04b6a8a229f6cf76a14b3 (DEFAULT_TABLE table=test-table [id=4183f02bc39c45ad8bcb0842b73d5e1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:14:10.284216 7776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:10.274439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33567 (local address 127.0.0.1:55940)
0924 04:14:10.279380 (+ 4941us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:10.279393 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:10.279541 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:10.280287 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:10.280290 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:10.280307 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:14:10.280379 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:10.280384 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.282101 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.282103 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:10.282816 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:10.282822 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.283891 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.283894 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:10.283953 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:10.283970 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 04:14:10.284031 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 04:14:10.284097 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4880,"thread_start_us":82,"threads_started":1}
I20260924 04:14:10.284291 7778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:10.274586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55940 (local address 127.0.98.2:33567)
0924 04:14:10.275184 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:10.275196 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:10.279477 (+ 4281us) server_negotiation.cc:407] Connection header received
0924 04:14:10.279673 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:10.279679 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:10.279955 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:10.280079 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:14:10.280500 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.281993 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.282925 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.283644 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.283734 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:10.283844 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:10.284076 (+ 232us) server_negotiation.cc:299] Negotiation successful
0924 04:14:10.284164 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":112,"threads_started":1}
I20260924 04:14:10.284689 7679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 862e4fb4e4d04b6a8a229f6cf76a14b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:14:10.285003 7406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 862e4fb4e4d04b6a8a229f6cf76a14b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:14:10.286504 7543 tablet_service.cc:1467] Processing CreateTablet for tablet 862e4fb4e4d04b6a8a229f6cf76a14b3 (DEFAULT_TABLE table=test-table [id=4183f02bc39c45ad8bcb0842b73d5e1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:14:10.287369 7543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 862e4fb4e4d04b6a8a229f6cf76a14b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:14:10.292908 7780 tablet_bootstrap.cc:492] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 7627c004b34c46e89a393afbfb63f656: Bootstrap starting.
I20260924 04:14:10.293810 7782 tablet_bootstrap.cc:492] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 12f50fe1de024fe1b72357e102c83015: Bootstrap starting.
I20260924 04:14:10.295756 7781 tablet_bootstrap.cc:492] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 34d2513e892f45b781f55ccdeae8ecd7: Bootstrap starting.
I20260924 04:14:10.295994 7780 tablet_bootstrap.cc:654] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 7627c004b34c46e89a393afbfb63f656: Neither blocks nor log segments found. Creating new log.
I20260924 04:14:10.296053 7782 tablet_bootstrap.cc:654] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 12f50fe1de024fe1b72357e102c83015: Neither blocks nor log segments found. Creating new log.
I20260924 04:14:10.296657 7782 log.cc:824] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 12f50fe1de024fe1b72357e102c83015: Log is configured to *not* fsync() on all Append() calls
I20260924 04:14:10.296679 7780 log.cc:824] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 7627c004b34c46e89a393afbfb63f656: Log is configured to *not* fsync() on all Append() calls
I20260924 04:14:10.298514 7780 tablet_bootstrap.cc:492] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 7627c004b34c46e89a393afbfb63f656: No bootstrap required, opened a new log
I20260924 04:14:10.298514 7782 tablet_bootstrap.cc:492] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 12f50fe1de024fe1b72357e102c83015: No bootstrap required, opened a new log
I20260924 04:14:10.298547 7781 tablet_bootstrap.cc:654] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 34d2513e892f45b781f55ccdeae8ecd7: Neither blocks nor log segments found. Creating new log.
I20260924 04:14:10.298696 7780 ts_tablet_manager.cc:1397] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 7627c004b34c46e89a393afbfb63f656: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260924 04:14:10.298696 7782 ts_tablet_manager.cc:1397] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 12f50fe1de024fe1b72357e102c83015: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 04:14:10.299197 7781 log.cc:824] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 34d2513e892f45b781f55ccdeae8ecd7: Log is configured to *not* fsync() on all Append() calls
I20260924 04:14:10.301208 7781 tablet_bootstrap.cc:492] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 34d2513e892f45b781f55ccdeae8ecd7: No bootstrap required, opened a new log
I20260924 04:14:10.301378 7781 ts_tablet_manager.cc:1397] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 34d2513e892f45b781f55ccdeae8ecd7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:14:10.303133 7780 raft_consensus.cc:348] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 7627c004b34c46e89a393afbfb63f656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7627c004b34c46e89a393afbfb63f656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41945 } } peers { permanent_uuid: "34d2513e892f45b781f55ccdeae8ecd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36843 } } peers { permanent_uuid: "12f50fe1de024fe1b72357e102c83015" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33567 } }
I20260924 04:14:10.303133 7782 raft_consensus.cc:348] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 12f50fe1de024fe1b72357e102c83015 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7627c004b34c46e89a393afbfb63f656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41945 } } peers { permanent_uuid: "34d2513e892f45b781f55ccdeae8ecd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36843 } } peers { permanent_uuid: "12f50fe1de024fe1b72357e102c83015" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33567 } }
I20260924 04:14:10.303422 7780 raft_consensus.cc:374] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 7627c004b34c46e89a393afbfb63f656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:14:10.303422 7782 raft_consensus.cc:374] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 12f50fe1de024fe1b72357e102c83015 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:14:10.303483 7780 raft_consensus.cc:729] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 7627c004b34c46e89a393afbfb63f656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7627c004b34c46e89a393afbfb63f656, State: Initialized, Role: FOLLOWER
I20260924 04:14:10.303483 7782 raft_consensus.cc:729] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 12f50fe1de024fe1b72357e102c83015 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12f50fe1de024fe1b72357e102c83015, State: Initialized, Role: FOLLOWER
I20260924 04:14:10.303961 7782 consensus_queue.cc:260] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 12f50fe1de024fe1b72357e102c83015 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7627c004b34c46e89a393afbfb63f656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41945 } } peers { permanent_uuid: "34d2513e892f45b781f55ccdeae8ecd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36843 } } peers { permanent_uuid: "12f50fe1de024fe1b72357e102c83015" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33567 } }
I20260924 04:14:10.303961 7780 consensus_queue.cc:260] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 7627c004b34c46e89a393afbfb63f656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7627c004b34c46e89a393afbfb63f656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41945 } } peers { permanent_uuid: "34d2513e892f45b781f55ccdeae8ecd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36843 } } peers { permanent_uuid: "12f50fe1de024fe1b72357e102c83015" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33567 } }
I20260924 04:14:10.304802 7610 heartbeater.cc:503] Master 127.0.98.62:34999 was elected leader, sending a full tablet report...
I20260924 04:14:10.304943 7782 ts_tablet_manager.cc:1428] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 12f50fe1de024fe1b72357e102c83015: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 04:14:10.304937 7780 ts_tablet_manager.cc:1428] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 7627c004b34c46e89a393afbfb63f656: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 04:14:10.305163 7747 heartbeater.cc:503] Master 127.0.98.62:34999 was elected leader, sending a full tablet report...
I20260924 04:14:10.309659 7781 raft_consensus.cc:348] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 34d2513e892f45b781f55ccdeae8ecd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7627c004b34c46e89a393afbfb63f656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41945 } } peers { permanent_uuid: "34d2513e892f45b781f55ccdeae8ecd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36843 } } peers { permanent_uuid: "12f50fe1de024fe1b72357e102c83015" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33567 } }
I20260924 04:14:10.309887 7781 raft_consensus.cc:374] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 34d2513e892f45b781f55ccdeae8ecd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:14:10.309943 7781 raft_consensus.cc:729] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 34d2513e892f45b781f55ccdeae8ecd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34d2513e892f45b781f55ccdeae8ecd7, State: Initialized, Role: FOLLOWER
I20260924 04:14:10.310322 7781 consensus_queue.cc:260] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 34d2513e892f45b781f55ccdeae8ecd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7627c004b34c46e89a393afbfb63f656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41945 } } peers { permanent_uuid: "34d2513e892f45b781f55ccdeae8ecd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36843 } } peers { permanent_uuid: "12f50fe1de024fe1b72357e102c83015" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33567 } }
I20260924 04:14:10.310917 7473 heartbeater.cc:503] Master 127.0.98.62:34999 was elected leader, sending a full tablet report...
I20260924 04:14:10.311252 7781 ts_tablet_manager.cc:1428] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 34d2513e892f45b781f55ccdeae8ecd7: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
W20260924 04:14:10.342803 7475 tablet.cc:2369] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 34d2513e892f45b781f55ccdeae8ecd7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:14:10.480697 7749 tablet.cc:2369] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 7627c004b34c46e89a393afbfb63f656: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:14:10.496948 7787 raft_consensus.cc:482] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 12f50fe1de024fe1b72357e102c83015 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:14:10.497144 7787 raft_consensus.cc:504] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 12f50fe1de024fe1b72357e102c83015 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7627c004b34c46e89a393afbfb63f656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41945 } } peers { permanent_uuid: "34d2513e892f45b781f55ccdeae8ecd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36843 } } peers { permanent_uuid: "12f50fe1de024fe1b72357e102c83015" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33567 } }
I20260924 04:14:10.498178 7787 leader_election.cc:288] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 12f50fe1de024fe1b72357e102c83015 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7627c004b34c46e89a393afbfb63f656 (127.0.98.3:41945), 34d2513e892f45b781f55ccdeae8ecd7 (127.0.98.1:36843)
I20260924 04:14:10.502655 7613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:10.498421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36843 (local address 127.0.98.2:50649)
0924 04:14:10.498803 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:10.498817 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:10.498920 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:10.499148 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:10.499151 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:10.499165 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:14:10.499277 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:10.499284 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.499973 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.499976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:10.501317 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:10.501327 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.502221 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.502225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:10.502318 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:10.502361 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 04:14:10.502432 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 04:14:10.502503 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":321}
I20260924 04:14:10.502636 7779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:10.498512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50649 (local address 127.0.98.1:36843)
0924 04:14:10.498632 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:10.498637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:10.498904 (+ 267us) server_negotiation.cc:407] Connection header received
0924 04:14:10.499020 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:10.499023 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:10.499071 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:10.499140 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:14:10.499383 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.499898 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.501438 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.502096 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.502199 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:10.502262 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:10.502447 (+ 185us) server_negotiation.cc:299] Negotiation successful
0924 04:14:10.502521 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 04:14:10.502702 7614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:10.498421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41945 (local address 127.0.98.2:57811)
0924 04:14:10.498525 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:10.498539 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:10.498632 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:10.498938 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:10.498941 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:10.498958 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:14:10.499081 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:10.499088 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.499926 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.499929 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:10.501316 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:10.501327 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.502333 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.502336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:10.502429 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:10.502450 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 04:14:10.502515 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 04:14:10.502581 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":84}
I20260924 04:14:10.503186 7775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:10.498582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57811 (local address 127.0.98.3:41945)
0924 04:14:10.498698 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:10.498701 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:10.498714 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:14:10.498770 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:10.498773 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:10.498824 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:10.498904 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:14:10.499180 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.499828 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.501519 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.502097 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.502199 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:10.502262 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:10.503004 (+ 742us) server_negotiation.cc:299] Negotiation successful
0924 04:14:10.503075 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 04:14:10.503419 7426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "862e4fb4e4d04b6a8a229f6cf76a14b3" candidate_uuid: "12f50fe1de024fe1b72357e102c83015" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34d2513e892f45b781f55ccdeae8ecd7" is_pre_election: true
I20260924 04:14:10.503753 7426 raft_consensus.cc:2454] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 34d2513e892f45b781f55ccdeae8ecd7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12f50fe1de024fe1b72357e102c83015 in term 0.
I20260924 04:14:10.503751 7699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "862e4fb4e4d04b6a8a229f6cf76a14b3" candidate_uuid: "12f50fe1de024fe1b72357e102c83015" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7627c004b34c46e89a393afbfb63f656" is_pre_election: true
I20260924 04:14:10.504014 7699 raft_consensus.cc:2454] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 7627c004b34c46e89a393afbfb63f656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12f50fe1de024fe1b72357e102c83015 in term 0.
I20260924 04:14:10.504208 7497 leader_election.cc:302] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 12f50fe1de024fe1b72357e102c83015 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12f50fe1de024fe1b72357e102c83015, 34d2513e892f45b781f55ccdeae8ecd7; no voters:
I20260924 04:14:10.504475 7787 raft_consensus.cc:2781] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 12f50fe1de024fe1b72357e102c83015 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:14:10.504549 7787 raft_consensus.cc:482] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 12f50fe1de024fe1b72357e102c83015 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:14:10.504590 7787 raft_consensus.cc:3037] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 12f50fe1de024fe1b72357e102c83015 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:14:10.505580 7787 raft_consensus.cc:504] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 12f50fe1de024fe1b72357e102c83015 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7627c004b34c46e89a393afbfb63f656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41945 } } peers { permanent_uuid: "34d2513e892f45b781f55ccdeae8ecd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36843 } } peers { permanent_uuid: "12f50fe1de024fe1b72357e102c83015" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33567 } }
I20260924 04:14:10.505903 7787 leader_election.cc:288] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 12f50fe1de024fe1b72357e102c83015 [CANDIDATE]: Term 1 election: Requested vote from peers 7627c004b34c46e89a393afbfb63f656 (127.0.98.3:41945), 34d2513e892f45b781f55ccdeae8ecd7 (127.0.98.1:36843)
I20260924 04:14:10.506204 7699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "862e4fb4e4d04b6a8a229f6cf76a14b3" candidate_uuid: "12f50fe1de024fe1b72357e102c83015" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7627c004b34c46e89a393afbfb63f656"
I20260924 04:14:10.506211 7426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "862e4fb4e4d04b6a8a229f6cf76a14b3" candidate_uuid: "12f50fe1de024fe1b72357e102c83015" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34d2513e892f45b781f55ccdeae8ecd7"
I20260924 04:14:10.506307 7699 raft_consensus.cc:3037] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 7627c004b34c46e89a393afbfb63f656 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:14:10.506323 7426 raft_consensus.cc:3037] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 34d2513e892f45b781f55ccdeae8ecd7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:14:10.507290 7699 raft_consensus.cc:2454] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 7627c004b34c46e89a393afbfb63f656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12f50fe1de024fe1b72357e102c83015 in term 1.
I20260924 04:14:10.507292 7426 raft_consensus.cc:2454] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 34d2513e892f45b781f55ccdeae8ecd7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12f50fe1de024fe1b72357e102c83015 in term 1.
I20260924 04:14:10.507587 7499 leader_election.cc:302] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 12f50fe1de024fe1b72357e102c83015 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 12f50fe1de024fe1b72357e102c83015, 34d2513e892f45b781f55ccdeae8ecd7, 7627c004b34c46e89a393afbfb63f656; no voters:
I20260924 04:14:10.507761 7787 raft_consensus.cc:2781] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 12f50fe1de024fe1b72357e102c83015 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:14:10.507944 7787 raft_consensus.cc:686] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 12f50fe1de024fe1b72357e102c83015 [term 1 LEADER]: Becoming Leader. State: Replica: 12f50fe1de024fe1b72357e102c83015, State: Running, Role: LEADER
I20260924 04:14:10.508276 7787 consensus_queue.cc:237] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 12f50fe1de024fe1b72357e102c83015 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7627c004b34c46e89a393afbfb63f656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41945 } } peers { permanent_uuid: "34d2513e892f45b781f55ccdeae8ecd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36843 } } peers { permanent_uuid: "12f50fe1de024fe1b72357e102c83015" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33567 } }
I20260924 04:14:10.511615 6502 catalog_manager.cc:5495] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 12f50fe1de024fe1b72357e102c83015 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12f50fe1de024fe1b72357e102c83015 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "12f50fe1de024fe1b72357e102c83015" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7627c004b34c46e89a393afbfb63f656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34d2513e892f45b781f55ccdeae8ecd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12f50fe1de024fe1b72357e102c83015" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33567 } health_report { overall_health: HEALTHY } } }
W20260924 04:14:10.534807 7612 tablet.cc:2369] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 12f50fe1de024fe1b72357e102c83015: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:14:10.566766 7778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:10.563703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55952 (local address 127.0.98.2:33567)
0924 04:14:10.563843 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:10.563848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:10.563862 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:14:10.563943 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:10.563945 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:10.563999 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:10.564081 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:14:10.564262 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.564749 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.565519 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.565726 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.565815 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:10.566023 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:14:10.566433 (+ 410us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:14:10.566518 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:10.566563 (+ 45us) server_negotiation.cc:299] Negotiation successful
0924 04:14:10.566614 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 04:14:10.572352 7426 raft_consensus.cc:1261] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 34d2513e892f45b781f55ccdeae8ecd7 [term 1 FOLLOWER]: Refusing update from remote peer 12f50fe1de024fe1b72357e102c83015: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:14:10.572358 7699 raft_consensus.cc:1261] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 7627c004b34c46e89a393afbfb63f656 [term 1 FOLLOWER]: Refusing update from remote peer 12f50fe1de024fe1b72357e102c83015: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:14:10.573026 7787 consensus_queue.cc:1035] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 12f50fe1de024fe1b72357e102c83015 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34d2513e892f45b781f55ccdeae8ecd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:14:10.573194 7789 consensus_queue.cc:1035] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 12f50fe1de024fe1b72357e102c83015 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7627c004b34c46e89a393afbfb63f656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:14:10.584044 7794 mvcc.cc:205] Tried to move back new op lower bound from 7332754434337280000 to 7332754434084528128. Current Snapshot: MvccSnapshot[applied={T|T < 7332754434337280000}]
I20260924 04:14:10.590409 6503 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:14:10.590559 6503 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:14:10.594580 6503 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c0d01bc511ef45b399351d4e77c4c85d: Sending DeleteTablet for 3 replicas of tablet 862e4fb4e4d04b6a8a229f6cf76a14b3
I20260924 04:14:10.595346 7679 tablet_service.cc:1514] Processing DeleteTablet for tablet 862e4fb4e4d04b6a8a229f6cf76a14b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:14:10 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49172
I20260924 04:14:10.595513 7406 tablet_service.cc:1514] Processing DeleteTablet for tablet 862e4fb4e4d04b6a8a229f6cf76a14b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:14:10 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58116
I20260924 04:14:10.595952 7802 tablet_replica.cc:330] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 34d2513e892f45b781f55ccdeae8ecd7: stopping tablet replica
I20260924 04:14:10.596259 7802 raft_consensus.cc:2229] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 34d2513e892f45b781f55ccdeae8ecd7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:14:10.596568 7802 raft_consensus.cc:2258] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 34d2513e892f45b781f55ccdeae8ecd7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:14:10.597340 7543 tablet_service.cc:1514] Processing DeleteTablet for tablet 862e4fb4e4d04b6a8a229f6cf76a14b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:14:10 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55940
I20260924 04:14:10.598157 7802 ts_tablet_manager.cc:1905] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 34d2513e892f45b781f55ccdeae8ecd7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:14:10.598170 7801 tablet_replica.cc:330] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 7627c004b34c46e89a393afbfb63f656: stopping tablet replica
I20260924 04:14:10.598429 7804 tablet_replica.cc:330] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 12f50fe1de024fe1b72357e102c83015: stopping tablet replica
I20260924 04:14:10.599617 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 6471
I20260924 04:14:10.601725 7802 ts_tablet_manager.cc:1918] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 34d2513e892f45b781f55ccdeae8ecd7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:14:10.601828 7802 log.cc:1196] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 34d2513e892f45b781f55ccdeae8ecd7: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-0/wal/wals/862e4fb4e4d04b6a8a229f6cf76a14b3
I20260924 04:14:10.602093 7802 ts_tablet_manager.cc:1939] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 34d2513e892f45b781f55ccdeae8ecd7: Deleting consensus metadata
W20260924 04:14:10.606099 7361 connection.cc:535] server connection from 127.0.0.1:58116 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 04:14:10.607564 7804 raft_consensus.cc:2229] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 12f50fe1de024fe1b72357e102c83015 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:14:10.607928 7801 raft_consensus.cc:2229] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 7627c004b34c46e89a393afbfb63f656 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:14:10.608335 7801 raft_consensus.cc:2258] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 7627c004b34c46e89a393afbfb63f656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:14:10.608691 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:10.597864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54842 (local address 127.0.98.61:45405)
0924 04:14:10.598066 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:10.598070 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:10.598084 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:14:10.598123 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:10.598126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:10.598169 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:10.598243 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:10.600595 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.601318 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.602754 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.603088 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.603161 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:10.605128 (+ 1967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:10.605132 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:10.605135 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:10.605161 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:10.606858 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:10.607353 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:10.607357 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:10.607359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:10.607449 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:10.607744 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:10.607747 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:10.607748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:10.607967 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:10.608085 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:10.608414 (+ 329us) server_negotiation.cc:299] Negotiation successful
0924 04:14:10.608530 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
I20260924 04:14:10.608719 7804 raft_consensus.cc:2258] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 12f50fe1de024fe1b72357e102c83015 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:14:10.610873 7801 ts_tablet_manager.cc:1905] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 7627c004b34c46e89a393afbfb63f656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:14:10.611423 7804 ts_tablet_manager.cc:1905] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 12f50fe1de024fe1b72357e102c83015: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:14:10.613608 7801 ts_tablet_manager.cc:1918] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 7627c004b34c46e89a393afbfb63f656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:14:10.613752 7801 log.cc:1196] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 7627c004b34c46e89a393afbfb63f656: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-2/wal/wals/862e4fb4e4d04b6a8a229f6cf76a14b3
I20260924 04:14:10.613994 7804 ts_tablet_manager.cc:1918] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 12f50fe1de024fe1b72357e102c83015: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:14:10.614090 7804 log.cc:1196] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 12f50fe1de024fe1b72357e102c83015: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-1/wal/wals/862e4fb4e4d04b6a8a229f6cf76a14b3
I20260924 04:14:10.614207 7801 ts_tablet_manager.cc:1939] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 7627c004b34c46e89a393afbfb63f656: Deleting consensus metadata
I20260924 04:14:10.614398 7804 ts_tablet_manager.cc:1939] T 862e4fb4e4d04b6a8a229f6cf76a14b3 P 12f50fe1de024fe1b72357e102c83015: Deleting consensus metadata
W20260924 04:14:10.615391 7635 connection.cc:441] server connection from 127.0.0.1:49172 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49172 (request call id 1) could send its response
W20260924 04:14:10.618515 7497 connection.cc:441] server connection from 127.0.0.1:55940 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55940 (request call id 1) could send its response
I20260924 04:14:10.620658 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:10.610352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54858 (local address 127.0.98.61:45405)
0924 04:14:10.610450 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:10.610454 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:10.611826 (+ 1372us) server_negotiation.cc:407] Connection header received
0924 04:14:10.611991 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:10.611994 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:10.612035 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:10.612078 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:10.613011 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.613784 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.615129 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.615495 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.615579 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:10.617271 (+ 1692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:10.617275 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:10.617278 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:10.617302 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:10.618864 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:10.619431 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:10.619434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:10.619436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:10.619515 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:10.619783 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:10.619786 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:10.619788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:10.619988 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:10.620082 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:10.620390 (+ 308us) server_negotiation.cc:299] Negotiation successful
0924 04:14:10.620500 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 04:14:10.623808 7761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:10.597960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41700 (local address 127.0.98.60:36165)
0924 04:14:10.598395 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:10.598399 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:10.610426 (+ 12027us) server_negotiation.cc:407] Connection header received
0924 04:14:10.615420 (+ 4994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:10.615424 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:10.615478 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:10.615528 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:10.616357 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.617138 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.618034 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.618772 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.618884 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:10.620434 (+ 1550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:10.620438 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:10.620440 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:10.620466 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:10.621993 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:10.622377 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:10.622381 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:10.622382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:10.622460 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:10.622735 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:10.622739 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:10.622740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:10.622952 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:10.623077 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:10.623300 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 04:14:10.623419 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364}
I20260924 04:14:10.625909 7479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:10.610894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41712 (local address 127.0.98.60:36165)
0924 04:14:10.611001 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:10.611005 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:10.618294 (+ 7289us) server_negotiation.cc:407] Connection header received
0924 04:14:10.618332 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:10.618335 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:10.618384 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:10.618448 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:10.619400 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.620155 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.621336 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:10.621650 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:10.621722 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:10.623309 (+ 1587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:10.623313 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:10.623315 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:10.623342 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:10.624548 (+ 1206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:10.624897 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:10.624901 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:10.624902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:10.624976 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:10.625211 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:10.625214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:10.625216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:10.625394 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:10.625501 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:10.625661 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 04:14:10.625760 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":500,"server-negotiator.queue_time_us":43}
I20260924 04:14:11.585596 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:11.585152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34999 (local address 127.0.98.3:52051)
0924 04:14:11.585393 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:11.585484 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":66,"threads_started":1}
W20260924 04:14:11.586164 7747 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34999 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:34999: connect: Connection refused (error 111)
I20260924 04:14:11.588758 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:11.588418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34999 (local address 127.0.98.1:35429)
0924 04:14:11.588615 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:11.588683 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":59,"threads_started":1}
W20260924 04:14:11.589053 7473 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34999 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:34999: connect: Connection refused (error 111)
I20260924 04:14:11.596903 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:11.596505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34999 (local address 127.0.98.2:47065)
0924 04:14:11.596747 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:11.596822 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":73,"threads_started":1}
W20260924 04:14:11.597162 7610 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34999 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:34999: connect: Connection refused (error 111)
I20260924 04:14:12.095784 7815 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0d01bc511ef45b399351d4e77c4c85d)
I20260924 04:14:12.096016 7815 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c0d01bc511ef45b399351d4e77c4c85d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } } peers { permanent_uuid: "15bf6683fb84414380dea83f7e7f93d8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45405 } attrs { promote: false } } peers { permanent_uuid: "86c8e5a65b994a6d91ae2f5ff4fc249d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36165 } attrs { promote: false } }
I20260924 04:14:12.097247 7815 leader_election.cc:288] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0d01bc511ef45b399351d4e77c4c85d (127.0.98.62:34999), 86c8e5a65b994a6d91ae2f5ff4fc249d (127.0.98.60:36165)
I20260924 04:14:12.098003 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:12.097452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34999 (local address 127.0.0.1:45410)
0924 04:14:12.097810 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:12.097897 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":67,"threads_started":1}
W20260924 04:14:12.098191 7090 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34999: connect: Connection refused (error 111)
I20260924 04:14:12.098543 7816 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0d01bc511ef45b399351d4e77c4c85d)
I20260924 04:14:12.098910 7816 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c0d01bc511ef45b399351d4e77c4c85d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } } peers { permanent_uuid: "15bf6683fb84414380dea83f7e7f93d8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45405 } attrs { promote: false } } peers { permanent_uuid: "86c8e5a65b994a6d91ae2f5ff4fc249d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36165 } attrs { promote: false } }
I20260924 04:14:12.098971 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:12.098758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34999 (local address 127.0.0.1:45426)
0924 04:14:12.098859 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:12.098907 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260924 04:14:12.099081 7090 leader_election.cc:334] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c0d01bc511ef45b399351d4e77c4c85d (127.0.98.62:34999): Network error: Client connection negotiation failed: client connection to 127.0.98.62:34999: connect: Connection refused (error 111)
I20260924 04:14:12.100081 7816 leader_election.cc:288] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0d01bc511ef45b399351d4e77c4c85d (127.0.98.62:34999), 15bf6683fb84414380dea83f7e7f93d8 (127.0.98.61:45405)
I20260924 04:14:12.100517 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:12.100143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34999 (local address 127.0.0.1:45438)
0924 04:14:12.100319 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:12.100438 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":65,"threads_started":1}
W20260924 04:14:12.100669 7280 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34999: connect: Connection refused (error 111)
I20260924 04:14:12.101404 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:12.101204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34999 (local address 127.0.0.1:45452)
0924 04:14:12.101294 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:12.101342 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 04:14:12.101526 7280 leader_election.cc:334] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c0d01bc511ef45b399351d4e77c4c85d (127.0.98.62:34999): Network error: Client connection negotiation failed: client connection to 127.0.98.62:34999: connect: Connection refused (error 111)
I20260924 04:14:12.102015 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:12.097748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36165 (local address 127.0.0.1:41714)
0924 04:14:12.097943 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:12.097956 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:12.098063 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:12.098447 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:12.098450 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:12.098471 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:14:12.098573 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:12.098580 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:12.099543 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:12.099546 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:12.100900 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:12.100909 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:12.101642 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:12.101646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:12.101736 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:12.101758 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 04:14:12.101825 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 04:14:12.101904 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":60,"threads_started":1}
I20260924 04:14:12.102015 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:12.097923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41714 (local address 127.0.98.60:36165)
0924 04:14:12.098216 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:12.098220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:12.098236 (+ 16us) server_negotiation.cc:407] Connection header received
0924 04:14:12.098291 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:12.098294 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:12.098342 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:12.098438 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:14:12.098707 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:12.099442 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:12.101001 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:12.101426 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:12.101607 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:12.101752 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:12.101836 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 04:14:12.101904 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":229,"thread_start_us":59,"threads_started":1}
I20260924 04:14:12.102458 7297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15bf6683fb84414380dea83f7e7f93d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "86c8e5a65b994a6d91ae2f5ff4fc249d" is_pre_election: true
I20260924 04:14:12.102697 7297 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15bf6683fb84414380dea83f7e7f93d8 in term 1.
I20260924 04:14:12.103000 7090 leader_election.cc:302] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 15bf6683fb84414380dea83f7e7f93d8, 86c8e5a65b994a6d91ae2f5ff4fc249d; no voters: c0d01bc511ef45b399351d4e77c4c85d
I20260924 04:14:12.103202 7815 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 04:14:12.103298 7815 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c0d01bc511ef45b399351d4e77c4c85d)
I20260924 04:14:12.103354 7815 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8 [term 1 FOLLOWER]: Advancing to term 2
I20260924 04:14:12.104579 7815 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c0d01bc511ef45b399351d4e77c4c85d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } } peers { permanent_uuid: "15bf6683fb84414380dea83f7e7f93d8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45405 } attrs { promote: false } } peers { permanent_uuid: "86c8e5a65b994a6d91ae2f5ff4fc249d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36165 } attrs { promote: false } }
I20260924 04:14:12.104974 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:12.100491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54868 (local address 127.0.98.61:45405)
0924 04:14:12.101326 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:12.101337 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:12.101349 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:14:12.101386 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:12.101389 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:12.101430 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:12.101505 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:14:12.102155 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:12.102757 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:12.104079 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:12.104473 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:12.104549 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:12.104592 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:12.104830 (+ 238us) server_negotiation.cc:299] Negotiation successful
0924 04:14:12.104888 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":78,"threads_started":1}
I20260924 04:14:12.105022 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:12.100422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45405 (local address 127.0.0.1:54868)
0924 04:14:12.100648 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:12.100660 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:12.100736 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:12.101603 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:12.101605 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:12.101622 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:14:12.101758 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:12.101763 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:12.102843 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:12.102847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:12.103961 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:12.103972 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:12.104590 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:12.104594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:12.104690 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:12.104720 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 04:14:12.104815 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 04:14:12.104886 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":19,"thread_start_us":59,"threads_started":1}
I20260924 04:14:12.105456 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:12.105104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34999 (local address 127.0.0.1:45454)
0924 04:14:12.105373 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:12.105412 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217}
I20260924 04:14:12.105464 7113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "86c8e5a65b994a6d91ae2f5ff4fc249d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "15bf6683fb84414380dea83f7e7f93d8" is_pre_election: true
I20260924 04:14:12.105538 7297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15bf6683fb84414380dea83f7e7f93d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "86c8e5a65b994a6d91ae2f5ff4fc249d"
I20260924 04:14:12.105683 7297 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d [term 1 FOLLOWER]: Advancing to term 2
I20260924 04:14:12.105727 7113 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 86c8e5a65b994a6d91ae2f5ff4fc249d in current term 2: Already voted for candidate 15bf6683fb84414380dea83f7e7f93d8 in this term.
I20260924 04:14:12.106017 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:12.105838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34999 (local address 127.0.0.1:45460)
0924 04:14:12.105920 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:12.105965 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 04:14:12.106124 7090 leader_election.cc:334] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c0d01bc511ef45b399351d4e77c4c85d (127.0.98.62:34999): Network error: Client connection negotiation failed: client connection to 127.0.98.62:34999: connect: Connection refused (error 111)
I20260924 04:14:12.106755 7297 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15bf6683fb84414380dea83f7e7f93d8 in term 2.
I20260924 04:14:12.105000 7815 leader_election.cc:288] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8 [CANDIDATE]: Term 2 election: Requested vote from peers c0d01bc511ef45b399351d4e77c4c85d (127.0.98.62:34999), 86c8e5a65b994a6d91ae2f5ff4fc249d (127.0.98.60:36165)
I20260924 04:14:12.107021 7090 leader_election.cc:302] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 15bf6683fb84414380dea83f7e7f93d8, 86c8e5a65b994a6d91ae2f5ff4fc249d; no voters: c0d01bc511ef45b399351d4e77c4c85d
I20260924 04:14:12.107107 7280 leader_election.cc:302] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 86c8e5a65b994a6d91ae2f5ff4fc249d; no voters: 15bf6683fb84414380dea83f7e7f93d8, c0d01bc511ef45b399351d4e77c4c85d
I20260924 04:14:12.107163 7815 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 04:14:12.107388 7815 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8 [term 2 LEADER]: Becoming Leader. State: Replica: 15bf6683fb84414380dea83f7e7f93d8, State: Running, Role: LEADER
I20260924 04:14:12.107447 7816 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 04:14:12.107690 7815 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c0d01bc511ef45b399351d4e77c4c85d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } } peers { permanent_uuid: "15bf6683fb84414380dea83f7e7f93d8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45405 } attrs { promote: false } } peers { permanent_uuid: "86c8e5a65b994a6d91ae2f5ff4fc249d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36165 } attrs { promote: false } }
I20260924 04:14:12.109025 7825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15bf6683fb84414380dea83f7e7f93d8. Latest consensus state: current_term: 2 leader_uuid: "15bf6683fb84414380dea83f7e7f93d8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c0d01bc511ef45b399351d4e77c4c85d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } } peers { permanent_uuid: "15bf6683fb84414380dea83f7e7f93d8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45405 } attrs { promote: false } } peers { permanent_uuid: "86c8e5a65b994a6d91ae2f5ff4fc249d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36165 } attrs { promote: false } } }
I20260924 04:14:12.109133 7825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8 [sys.catalog]: This master's current role is: LEADER
I20260924 04:14:12.109390 7827 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:14:12.111483 7827 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:14:12.111872 7827 catalog_manager.cc:1259] Loaded cluster ID: 4c5e55d2a7c949cf9287fac82ca19300
I20260924 04:14:12.111915 7827 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:14:12.112416 7827 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:14:12.112756 7827 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8: Loaded TSK: 0
I20260924 04:14:12.113374 7827 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:14:12.143107 7102 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54858:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:14:12.146041 7102 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:14:12.148720 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:12.148491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34999 (local address 127.0.0.1:45478)
0924 04:14:12.148599 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:12.148654 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260924 04:14:12.148972 7297 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d [term 2 FOLLOWER]: Refusing update from remote peer 15bf6683fb84414380dea83f7e7f93d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260924 04:14:12.149345 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:12.149154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34999 (local address 127.0.0.1:45488)
0924 04:14:12.149245 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:12.149290 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 04:14:12.149432 7090 consensus_peers.cc:487] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8 -> Peer c0d01bc511ef45b399351d4e77c4c85d (127.0.98.62:34999): Couldn't send request to peer c0d01bc511ef45b399351d4e77c4c85d. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34999: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 04:14:12.149453 7825 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86c8e5a65b994a6d91ae2f5ff4fc249d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36165 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260924 04:14:12.150709 7816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15bf6683fb84414380dea83f7e7f93d8. Latest consensus state: current_term: 2 leader_uuid: "15bf6683fb84414380dea83f7e7f93d8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c0d01bc511ef45b399351d4e77c4c85d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } } peers { permanent_uuid: "15bf6683fb84414380dea83f7e7f93d8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45405 } attrs { promote: false } } peers { permanent_uuid: "86c8e5a65b994a6d91ae2f5ff4fc249d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36165 } attrs { promote: false } } }
I20260924 04:14:12.150846 7816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:14:12.151957 7825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "15bf6683fb84414380dea83f7e7f93d8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c0d01bc511ef45b399351d4e77c4c85d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } } peers { permanent_uuid: "15bf6683fb84414380dea83f7e7f93d8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45405 } attrs { promote: false } } peers { permanent_uuid: "86c8e5a65b994a6d91ae2f5ff4fc249d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36165 } attrs { promote: false } } }
I20260924 04:14:12.152069 7825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8 [sys.catalog]: This master's current role is: LEADER
I20260924 04:14:12.152237 7816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "15bf6683fb84414380dea83f7e7f93d8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c0d01bc511ef45b399351d4e77c4c85d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34999 } } peers { permanent_uuid: "15bf6683fb84414380dea83f7e7f93d8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45405 } attrs { promote: false } } peers { permanent_uuid: "86c8e5a65b994a6d91ae2f5ff4fc249d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36165 } attrs { promote: false } } }
I20260924 04:14:12.152352 7816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86c8e5a65b994a6d91ae2f5ff4fc249d [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:14:12.161347 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:12.157290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36843 (local address 127.0.0.1:58128)
0924 04:14:12.157427 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:12.157440 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:12.157507 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:12.157901 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:12.157904 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:12.157918 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:14:12.158015 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:12.158020 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:12.159045 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:12.159049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:12.160257 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:12.160266 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:12.160977 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:12.160981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:12.161059 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:12.161084 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 04:14:12.161152 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 04:14:12.161220 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"mutex_wait_us":21}
I20260924 04:14:12.161396 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:12.157409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58128 (local address 127.0.98.1:36843)
0924 04:14:12.157665 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:12.157670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:12.157684 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:14:12.157736 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:12.157739 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:12.157797 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:12.158117 (+ 320us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:14:12.158227 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:12.158943 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:12.160374 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:12.160872 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:12.160973 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:12.161037 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:12.161184 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 04:14:12.161262 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":89,"threads_started":1}
I20260924 04:14:12.161742 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:12.157723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33567 (local address 127.0.0.1:55960)
0924 04:14:12.158006 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:12.158017 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:12.158092 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:12.158613 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:12.158616 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:12.158631 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:14:12.158716 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:12.158721 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:12.159500 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:12.159508 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:12.160534 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:12.160540 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:12.161393 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:12.161396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:12.161479 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:12.161502 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 04:14:12.161560 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 04:14:12.161621 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":100,"threads_started":1}
I20260924 04:14:12.161895 7406 tablet_service.cc:1467] Processing CreateTablet for tablet b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a (DEFAULT_TABLE table=test-table [id=4f88a9cf94664db4af0dddfa6c981907]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:14:12.162101 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:12.157829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55960 (local address 127.0.98.2:33567)
0924 04:14:12.158337 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:12.158341 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:12.158354 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:14:12.158398 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:12.158401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:12.158443 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:12.158517 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:14:12.158825 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:12.159413 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:12.160630 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:12.161196 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:12.161284 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:12.161341 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:12.161896 (+ 555us) server_negotiation.cc:299] Negotiation successful
0924 04:14:12.161977 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":67,"threads_started":1}
I20260924 04:14:12.162189 7406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:14:12.162534 7543 tablet_service.cc:1467] Processing CreateTablet for tablet b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a (DEFAULT_TABLE table=test-table [id=4f88a9cf94664db4af0dddfa6c981907]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:14:12.162719 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:12.157477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41945 (local address 127.0.0.1:49186)
0924 04:14:12.157689 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:12.157701 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:12.157771 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:12.158411 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:12.158414 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:12.158430 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:14:12.158512 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:12.158518 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:12.159457 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:12.159460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:12.160264 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:12.160272 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:12.162383 (+ 2111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:12.162387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:12.162469 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:12.162489 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 04:14:12.162552 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 04:14:12.162613 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":25}
I20260924 04:14:12.162747 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:12.157922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49186 (local address 127.0.98.3:41945)
0924 04:14:12.158180 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:12.158184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:12.158198 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:14:12.158245 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:12.158247 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:12.158291 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:12.158610 (+ 319us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:14:12.158718 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:12.159359 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:12.161087 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:12.162096 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:12.162374 (+ 278us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:12.162431 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:12.162572 (+ 141us) server_negotiation.cc:299] Negotiation successful
0924 04:14:12.162636 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":84,"threads_started":1}
I20260924 04:14:12.162762 7543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:14:12.163663 7679 tablet_service.cc:1467] Processing CreateTablet for tablet b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a (DEFAULT_TABLE table=test-table [id=4f88a9cf94664db4af0dddfa6c981907]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:14:12.163918 7679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:14:12.164711 7837 tablet_bootstrap.cc:492] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 34d2513e892f45b781f55ccdeae8ecd7: Bootstrap starting.
I20260924 04:14:12.165360 7838 tablet_bootstrap.cc:492] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 12f50fe1de024fe1b72357e102c83015: Bootstrap starting.
I20260924 04:14:12.165632 7837 tablet_bootstrap.cc:654] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 34d2513e892f45b781f55ccdeae8ecd7: Neither blocks nor log segments found. Creating new log.
I20260924 04:14:12.166038 7839 tablet_bootstrap.cc:492] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 7627c004b34c46e89a393afbfb63f656: Bootstrap starting.
I20260924 04:14:12.166149 7838 tablet_bootstrap.cc:654] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 12f50fe1de024fe1b72357e102c83015: Neither blocks nor log segments found. Creating new log.
I20260924 04:14:12.166814 7839 tablet_bootstrap.cc:654] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 7627c004b34c46e89a393afbfb63f656: Neither blocks nor log segments found. Creating new log.
I20260924 04:14:12.167018 7837 tablet_bootstrap.cc:492] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 34d2513e892f45b781f55ccdeae8ecd7: No bootstrap required, opened a new log
I20260924 04:14:12.167094 7837 ts_tablet_manager.cc:1397] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 34d2513e892f45b781f55ccdeae8ecd7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:14:12.167570 7837 raft_consensus.cc:348] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 34d2513e892f45b781f55ccdeae8ecd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d2513e892f45b781f55ccdeae8ecd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36843 } } peers { permanent_uuid: "7627c004b34c46e89a393afbfb63f656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41945 } } peers { permanent_uuid: "12f50fe1de024fe1b72357e102c83015" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33567 } }
I20260924 04:14:12.167685 7837 raft_consensus.cc:374] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 34d2513e892f45b781f55ccdeae8ecd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:14:12.167717 7837 raft_consensus.cc:729] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 34d2513e892f45b781f55ccdeae8ecd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34d2513e892f45b781f55ccdeae8ecd7, State: Initialized, Role: FOLLOWER
I20260924 04:14:12.167798 7838 tablet_bootstrap.cc:492] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 12f50fe1de024fe1b72357e102c83015: No bootstrap required, opened a new log
I20260924 04:14:12.167870 7839 tablet_bootstrap.cc:492] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 7627c004b34c46e89a393afbfb63f656: No bootstrap required, opened a new log
I20260924 04:14:12.167873 7838 ts_tablet_manager.cc:1397] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 12f50fe1de024fe1b72357e102c83015: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 04:14:12.167838 7837 consensus_queue.cc:260] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 34d2513e892f45b781f55ccdeae8ecd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d2513e892f45b781f55ccdeae8ecd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36843 } } peers { permanent_uuid: "7627c004b34c46e89a393afbfb63f656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41945 } } peers { permanent_uuid: "12f50fe1de024fe1b72357e102c83015" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33567 } }
I20260924 04:14:12.167949 7839 ts_tablet_manager.cc:1397] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 7627c004b34c46e89a393afbfb63f656: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:14:12.168151 7837 ts_tablet_manager.cc:1428] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 34d2513e892f45b781f55ccdeae8ecd7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:14:12.168241 7838 raft_consensus.cc:348] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 12f50fe1de024fe1b72357e102c83015 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d2513e892f45b781f55ccdeae8ecd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36843 } } peers { permanent_uuid: "7627c004b34c46e89a393afbfb63f656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41945 } } peers { permanent_uuid: "12f50fe1de024fe1b72357e102c83015" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33567 } }
I20260924 04:14:12.168342 7838 raft_consensus.cc:374] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 12f50fe1de024fe1b72357e102c83015 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:14:12.168313 7839 raft_consensus.cc:348] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 7627c004b34c46e89a393afbfb63f656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d2513e892f45b781f55ccdeae8ecd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36843 } } peers { permanent_uuid: "7627c004b34c46e89a393afbfb63f656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41945 } } peers { permanent_uuid: "12f50fe1de024fe1b72357e102c83015" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33567 } }
I20260924 04:14:12.168375 7838 raft_consensus.cc:729] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 12f50fe1de024fe1b72357e102c83015 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12f50fe1de024fe1b72357e102c83015, State: Initialized, Role: FOLLOWER
I20260924 04:14:12.168404 7839 raft_consensus.cc:374] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 7627c004b34c46e89a393afbfb63f656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:14:12.168439 7839 raft_consensus.cc:729] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 7627c004b34c46e89a393afbfb63f656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7627c004b34c46e89a393afbfb63f656, State: Initialized, Role: FOLLOWER
I20260924 04:14:12.168480 7838 consensus_queue.cc:260] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 12f50fe1de024fe1b72357e102c83015 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d2513e892f45b781f55ccdeae8ecd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36843 } } peers { permanent_uuid: "7627c004b34c46e89a393afbfb63f656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41945 } } peers { permanent_uuid: "12f50fe1de024fe1b72357e102c83015" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33567 } }
I20260924 04:14:12.168548 7839 consensus_queue.cc:260] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 7627c004b34c46e89a393afbfb63f656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d2513e892f45b781f55ccdeae8ecd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36843 } } peers { permanent_uuid: "7627c004b34c46e89a393afbfb63f656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41945 } } peers { permanent_uuid: "12f50fe1de024fe1b72357e102c83015" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33567 } }
I20260924 04:14:12.168793 7838 ts_tablet_manager.cc:1428] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 12f50fe1de024fe1b72357e102c83015: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:14:12.168856 7839 ts_tablet_manager.cc:1428] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 7627c004b34c46e89a393afbfb63f656: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:14:12.171171 7474 heartbeater.cc:503] Master 127.0.98.61:45405 was elected leader, sending a full tablet report...
I20260924 04:14:12.171066 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:12.169681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34999 (local address 127.0.98.3:41359)
0924 04:14:12.170908 (+ 1227us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:12.170965 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1160,"thread_start_us":78,"threads_started":1}
I20260924 04:14:12.171773 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:12.168800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34999 (local address 127.0.98.1:49645)
0924 04:14:12.171631 (+ 2831us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:12.171685 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2768,"thread_start_us":77,"threads_started":1}
I20260924 04:14:12.172504 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:12.172145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34999 (local address 127.0.98.1:39959)
0924 04:14:12.172418 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:12.172458 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220}
I20260924 04:14:12.172806 7748 heartbeater.cc:503] Master 127.0.98.61:45405 was elected leader, sending a full tablet report...
I20260924 04:14:12.173174 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:12.172883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34999 (local address 127.0.98.3:48341)
0924 04:14:12.173065 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:12.173114 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260924 04:14:12.173241 7611 heartbeater.cc:503] Master 127.0.98.61:45405 was elected leader, sending a full tablet report...
I20260924 04:14:12.174372 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:12.169489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34999 (local address 127.0.98.2:60993)
0924 04:14:12.174226 (+ 4737us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:12.174285 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4121,"thread_start_us":80,"threads_started":1}
I20260924 04:14:12.174979 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:12.174773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34999 (local address 127.0.98.2:32901)
0924 04:14:12.174875 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:12.174923 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 04:14:12.246094 7845 raft_consensus.cc:482] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 7627c004b34c46e89a393afbfb63f656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:14:12.246255 7845 raft_consensus.cc:504] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 7627c004b34c46e89a393afbfb63f656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d2513e892f45b781f55ccdeae8ecd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36843 } } peers { permanent_uuid: "7627c004b34c46e89a393afbfb63f656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41945 } } peers { permanent_uuid: "12f50fe1de024fe1b72357e102c83015" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33567 } }
I20260924 04:14:12.247325 7845 leader_election.cc:288] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 7627c004b34c46e89a393afbfb63f656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 34d2513e892f45b781f55ccdeae8ecd7 (127.0.98.1:36843), 12f50fe1de024fe1b72357e102c83015 (127.0.98.2:33567)
I20260924 04:14:12.251953 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:12.247910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33231 (local address 127.0.98.2:33567)
0924 04:14:12.248025 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:12.248029 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:12.248173 (+ 144us) server_negotiation.cc:407] Connection header received
0924 04:14:12.248255 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:12.248257 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:12.248302 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:12.248373 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:14:12.248572 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:12.249263 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:12.250579 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:12.251184 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:12.251322 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:12.251399 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:12.251751 (+ 352us) server_negotiation.cc:299] Negotiation successful
0924 04:14:12.251840 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 04:14:12.251953 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:12.247629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52209 (local address 127.0.98.1:36843)
0924 04:14:12.247768 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:12.247772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:12.247789 (+ 17us) server_negotiation.cc:407] Connection header received
0924 04:14:12.247934 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:12.247938 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:12.247993 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:12.248066 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:14:12.248313 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:12.249083 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:12.250581 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:12.251201 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:12.251322 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:12.251399 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:12.251751 (+ 352us) server_negotiation.cc:299] Negotiation successful
0924 04:14:12.251824 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 04:14:12.252074 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:12.247779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33567 (local address 127.0.98.3:33231)
0924 04:14:12.248085 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:12.248097 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:12.248183 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:12.248390 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:12.248392 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:12.248407 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:14:12.248486 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:12.248490 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:12.249358 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:12.249361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:12.250470 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:12.250480 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:12.251336 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:12.251340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:12.251574 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:12.251634 (+ 60us) client_negotiation.cc:769] Sending connection context
0924 04:14:12.251737 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 04:14:12.251907 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":105,"thread_start_us":143,"threads_started":1}
I20260924 04:14:12.252074 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:12.247528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36843 (local address 127.0.98.3:52209)
0924 04:14:12.247659 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:12.247676 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:12.247819 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:12.248084 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:12.248087 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:12.248110 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:14:12.248218 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:12.248224 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:12.249186 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:12.249188 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:12.250470 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:12.250480 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:12.251344 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:12.251347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:12.251580 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:12.251661 (+ 81us) client_negotiation.cc:769] Sending connection context
0924 04:14:12.251736 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 04:14:12.251904 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":91}
I20260924 04:14:12.252599 7563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a" candidate_uuid: "7627c004b34c46e89a393afbfb63f656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12f50fe1de024fe1b72357e102c83015" is_pre_election: true
I20260924 04:14:12.252717 7426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a" candidate_uuid: "7627c004b34c46e89a393afbfb63f656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34d2513e892f45b781f55ccdeae8ecd7" is_pre_election: true
I20260924 04:14:12.252827 7563 raft_consensus.cc:2454] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 12f50fe1de024fe1b72357e102c83015 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7627c004b34c46e89a393afbfb63f656 in term 0.
I20260924 04:14:12.252875 7426 raft_consensus.cc:2454] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 34d2513e892f45b781f55ccdeae8ecd7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7627c004b34c46e89a393afbfb63f656 in term 0.
I20260924 04:14:12.253197 7633 leader_election.cc:302] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 7627c004b34c46e89a393afbfb63f656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12f50fe1de024fe1b72357e102c83015, 7627c004b34c46e89a393afbfb63f656; no voters:
I20260924 04:14:12.253436 7845 raft_consensus.cc:2781] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 7627c004b34c46e89a393afbfb63f656 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:14:12.253495 7845 raft_consensus.cc:482] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 7627c004b34c46e89a393afbfb63f656 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:14:12.253530 7845 raft_consensus.cc:3037] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 7627c004b34c46e89a393afbfb63f656 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:14:12.254771 7845 raft_consensus.cc:504] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 7627c004b34c46e89a393afbfb63f656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d2513e892f45b781f55ccdeae8ecd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36843 } } peers { permanent_uuid: "7627c004b34c46e89a393afbfb63f656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41945 } } peers { permanent_uuid: "12f50fe1de024fe1b72357e102c83015" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33567 } }
I20260924 04:14:12.255060 7845 leader_election.cc:288] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 7627c004b34c46e89a393afbfb63f656 [CANDIDATE]: Term 1 election: Requested vote from peers 34d2513e892f45b781f55ccdeae8ecd7 (127.0.98.1:36843), 12f50fe1de024fe1b72357e102c83015 (127.0.98.2:33567)
I20260924 04:14:12.255442 7563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a" candidate_uuid: "7627c004b34c46e89a393afbfb63f656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12f50fe1de024fe1b72357e102c83015"
I20260924 04:14:12.255443 7426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a" candidate_uuid: "7627c004b34c46e89a393afbfb63f656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34d2513e892f45b781f55ccdeae8ecd7"
I20260924 04:14:12.255563 7426 raft_consensus.cc:3037] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 34d2513e892f45b781f55ccdeae8ecd7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:14:12.255563 7563 raft_consensus.cc:3037] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 12f50fe1de024fe1b72357e102c83015 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:14:12.256625 7563 raft_consensus.cc:2454] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 12f50fe1de024fe1b72357e102c83015 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7627c004b34c46e89a393afbfb63f656 in term 1.
I20260924 04:14:12.256625 7426 raft_consensus.cc:2454] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 34d2513e892f45b781f55ccdeae8ecd7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7627c004b34c46e89a393afbfb63f656 in term 1.
I20260924 04:14:12.256868 7633 leader_election.cc:302] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 7627c004b34c46e89a393afbfb63f656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34d2513e892f45b781f55ccdeae8ecd7, 7627c004b34c46e89a393afbfb63f656; no voters:
I20260924 04:14:12.257014 7845 raft_consensus.cc:2781] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 7627c004b34c46e89a393afbfb63f656 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:14:12.257201 7845 raft_consensus.cc:686] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 7627c004b34c46e89a393afbfb63f656 [term 1 LEADER]: Becoming Leader. State: Replica: 7627c004b34c46e89a393afbfb63f656, State: Running, Role: LEADER
I20260924 04:14:12.257436 7845 consensus_queue.cc:237] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 7627c004b34c46e89a393afbfb63f656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d2513e892f45b781f55ccdeae8ecd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36843 } } peers { permanent_uuid: "7627c004b34c46e89a393afbfb63f656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41945 } } peers { permanent_uuid: "12f50fe1de024fe1b72357e102c83015" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33567 } }
I20260924 04:14:12.257958 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:12.257744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34999 (local address 127.0.98.3:47263)
0924 04:14:12.257851 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:12.257900 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 04:14:12.259670 7100 catalog_manager.cc:5495] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 7627c004b34c46e89a393afbfb63f656 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7627c004b34c46e89a393afbfb63f656 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "7627c004b34c46e89a393afbfb63f656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d2513e892f45b781f55ccdeae8ecd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7627c004b34c46e89a393afbfb63f656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12f50fe1de024fe1b72357e102c83015" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33567 } health_report { overall_health: UNKNOWN } } }
I20260924 04:14:12.285513 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:12.282245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49200 (local address 127.0.98.3:41945)
0924 04:14:12.282381 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:12.282385 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:12.282400 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:14:12.282545 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:12.282547 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:12.282601 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:12.282687 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:14:12.282902 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:12.283433 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:12.284214 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:12.284421 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:12.284502 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:12.284736 (+ 234us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:14:12.285163 (+ 427us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:14:12.285260 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:12.285334 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 04:14:12.285386 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 04:14:12.288049 7426 raft_consensus.cc:1261] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 34d2513e892f45b781f55ccdeae8ecd7 [term 1 FOLLOWER]: Refusing update from remote peer 7627c004b34c46e89a393afbfb63f656: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:14:12.288316 7563 raft_consensus.cc:1261] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 12f50fe1de024fe1b72357e102c83015 [term 1 FOLLOWER]: Refusing update from remote peer 7627c004b34c46e89a393afbfb63f656: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:14:12.288513 7850 consensus_queue.cc:1035] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 7627c004b34c46e89a393afbfb63f656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34d2513e892f45b781f55ccdeae8ecd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:14:12.288810 7845 consensus_queue.cc:1035] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 7627c004b34c46e89a393afbfb63f656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12f50fe1de024fe1b72357e102c83015" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:14:12.290647 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:12.290300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34999 (local address 127.0.98.1:48105)
0924 04:14:12.290514 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:12.290572 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
I20260924 04:14:12.291113 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:12.290904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34999 (local address 127.0.98.2:57849)
0924 04:14:12.290996 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:12.291046 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 04:14:12.292735 7855 mvcc.cc:205] Tried to move back new op lower bound from 7332754441367392256 to 7332754441248874496. Current Snapshot: MvccSnapshot[applied={T|T < 7332754441367392256}]
I20260924 04:14:12.293421 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:12.292963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34999 (local address 127.0.98.1:33777)
0924 04:14:12.293298 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:12.293357 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275}
I20260924 04:14:12.294839 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:12.294538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34999 (local address 127.0.98.3:57229)
0924 04:14:12.294726 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:12.294774 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260924 04:14:12.296269 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:12.296049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34999 (local address 127.0.98.3:36899)
0924 04:14:12.296165 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:12.296210 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260924 04:14:12.296351 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:12.295942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34999 (local address 127.0.98.2:58971)
0924 04:14:12.296233 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:12.296280 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230}
I20260924 04:14:12.299546 7857 mvcc.cc:205] Tried to move back new op lower bound from 7332754441367392256 to 7332754441248874496. Current Snapshot: MvccSnapshot[applied={T|T < 7332754441367392256 or (T in {7332754441367392256})}]
I20260924 04:14:12.301126 7100 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:14:12.301249 7100 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:14:12.304741 7100 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 15bf6683fb84414380dea83f7e7f93d8: Sending DeleteTablet for 3 replicas of tablet b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a
I20260924 04:14:12.305416 7406 tablet_service.cc:1514] Processing DeleteTablet for tablet b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:14:12 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:58128
I20260924 04:14:12.305644 7543 tablet_service.cc:1514] Processing DeleteTablet for tablet b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:14:12 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:55960
I20260924 04:14:12.305720 7679 tablet_service.cc:1514] Processing DeleteTablet for tablet b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:14:12 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:49186
I20260924 04:14:12.305948 7865 tablet_replica.cc:330] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 12f50fe1de024fe1b72357e102c83015: stopping tablet replica
I20260924 04:14:12.306110 7865 raft_consensus.cc:2229] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 12f50fe1de024fe1b72357e102c83015 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:14:12.306124 7864 tablet_replica.cc:330] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 34d2513e892f45b781f55ccdeae8ecd7: stopping tablet replica
I20260924 04:14:12.306232 7864 raft_consensus.cc:2229] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 34d2513e892f45b781f55ccdeae8ecd7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:14:12.306360 7865 raft_consensus.cc:2258] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 12f50fe1de024fe1b72357e102c83015 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:14:12.306367 7864 raft_consensus.cc:2258] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 34d2513e892f45b781f55ccdeae8ecd7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:14:12.306638 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 7247
I20260924 04:14:12.307739 7866 tablet_replica.cc:330] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 7627c004b34c46e89a393afbfb63f656: stopping tablet replica
I20260924 04:14:12.307868 7866 raft_consensus.cc:2229] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 7627c004b34c46e89a393afbfb63f656 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:14:12.308094 7866 raft_consensus.cc:2258] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 7627c004b34c46e89a393afbfb63f656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:14:12.308611 7866 ts_tablet_manager.cc:1905] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 7627c004b34c46e89a393afbfb63f656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:14:12.310621 7866 ts_tablet_manager.cc:1918] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 7627c004b34c46e89a393afbfb63f656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:14:12.310684 7866 log.cc:1196] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 7627c004b34c46e89a393afbfb63f656: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-2/wal/wals/b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a
I20260924 04:14:12.310899 7866 ts_tablet_manager.cc:1939] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 7627c004b34c46e89a393afbfb63f656: Deleting consensus metadata
I20260924 04:14:12.311542 7090 catalog_manager.cc:4841] TS 7627c004b34c46e89a393afbfb63f656 (127.0.98.3:41945): tablet b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a (table test-table [id=4f88a9cf94664db4af0dddfa6c981907]) successfully deleted
I20260924 04:14:12.313591 7865 ts_tablet_manager.cc:1905] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 12f50fe1de024fe1b72357e102c83015: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:14:12.315936 7865 ts_tablet_manager.cc:1918] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 12f50fe1de024fe1b72357e102c83015: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:14:12.316012 7865 log.cc:1196] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 12f50fe1de024fe1b72357e102c83015: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790223216265354-392-0/minicluster-data/ts-1/wal/wals/b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a
I20260924 04:14:12.316267 7865 ts_tablet_manager.cc:1939] T b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a P 12f50fe1de024fe1b72357e102c83015: Deleting consensus metadata
W20260924 04:14:12.316363 7817 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:12.313441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36843 (local address 127.0.0.1:58134)
0924 04:14:12.313556 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:12.313569 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:12.313639 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:12.316255 (+ 2616us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36843: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 04:14:12.316556 7088 catalog_manager.cc:4584] TS 34d2513e892f45b781f55ccdeae8ecd7 (127.0.98.1:36843): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36843: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 04:14:12.316810 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 7480
I20260924 04:14:12.316862 7088 catalog_manager.cc:4841] TS 12f50fe1de024fe1b72357e102c83015 (127.0.98.2:33567): tablet b53bdd3f2ae64a4f9ec4b9fbe8fc5b8a (table test-table [id=4f88a9cf94664db4af0dddfa6c981907]) successfully deleted
I20260924 04:14:12.322492 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 7616
I20260924 04:14:12.327504 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 6967
I20260924 04:14:12.331990 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 7075
2026-09-24T04:14:12Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5223 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:14:12 dist-test-slave-p401 krb5kdc[7869](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:14:12 dist-test-slave-p401 krb5kdc[7869](info): set up 2 sockets
Sep 24 04:14:12 dist-test-slave-p401 krb5kdc[7869](info): commencing operation
krb5kdc: starting...
W20260924 04:14:13.287689 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/krb5kdc/test-user.keytab.
Sep 24 04:14:13 dist-test-slave-p401 krb5kdc[7869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:14:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:14:13Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260924 04:14:13.402079 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39479
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39479
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:14:13.508100 7882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:14:13.508349 7882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:14:13.508396 7882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:14:13.512356 7882 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:14:13.512418 7882 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 04:14:13.512432 7882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:14:13.512449 7882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:14:13.512465 7882 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:14:13.512493 7882 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:14:13.517710 7882 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39479
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:14:13.518702 7882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:14:13.519737 7882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:14:13.527729 7890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:13.527750 7887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:13.527839 7882 server_base.cc:1034] running on GCE node
W20260924 04:14:13.527750 7888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:13.528481 7882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:14:13.529529 7882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:14:13.530678 7882 hybrid_clock.cc:648] HybridClock initialized: now 1790223253530662 us; error 29 us; skew 500 ppm
Sep 24 04:14:13 dist-test-slave-p401 krb5kdc[7869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223253, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:14:13.533209 7882 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260924 04:14:13.534003 7882 webserver.cc:460] Webserver started at http://127.0.98.62:34741/ using document root <none> and password file <none>
I20260924 04:14:13.534497 7882 fs_manager.cc:359] Metadata directory not provided
I20260924 04:14:13.534559 7882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:14:13.534746 7882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:14:13.536695 7882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/master-0/data/instance:
uuid: "32a4954b64f244c6aa3aee7bdcf49528"
format_stamp: "Formatted at 2026-09-24 04:14:13 on dist-test-slave-p401"
server_key: "155cdb08e61192d411f1683fa0d1cbac"
server_key_iv: "4b3e580bc3e6cb2ac61120a9ab20efb2"
server_key_version: "encryptionkey@0"
I20260924 04:14:13.537235 7882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/master-0/wal/instance:
uuid: "32a4954b64f244c6aa3aee7bdcf49528"
format_stamp: "Formatted at 2026-09-24 04:14:13 on dist-test-slave-p401"
server_key: "155cdb08e61192d411f1683fa0d1cbac"
server_key_iv: "4b3e580bc3e6cb2ac61120a9ab20efb2"
server_key_version: "encryptionkey@0"
I20260924 04:14:13.540616 7882 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260924 04:14:13.542781 7897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:13.543907 7882 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:14:13.544011 7882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/master-0/wal
uuid: "32a4954b64f244c6aa3aee7bdcf49528"
format_stamp: "Formatted at 2026-09-24 04:14:13 on dist-test-slave-p401"
server_key: "155cdb08e61192d411f1683fa0d1cbac"
server_key_iv: "4b3e580bc3e6cb2ac61120a9ab20efb2"
server_key_version: "encryptionkey@0"
I20260924 04:14:13.544092 7882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:14:13.560397 7882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:14:13.563010 7882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:14:13.563169 7882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:14:13.570377 7882 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39479
I20260924 04:14:13.570562 7959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39479 every 8 connection(s)
I20260924 04:14:13.571311 7882 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:14:13.573889 7960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:14:13.577891 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 7882
I20260924 04:14:13.578006 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/master-0/wal/instance
I20260924 04:14:13.578256 392 external_mini_cluster.cc:1428] Setting key 3f76f122cc3bb8fe3bdb42158afbe186
I20260924 04:14:13.579361 7960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32a4954b64f244c6aa3aee7bdcf49528: Bootstrap starting.
I20260924 04:14:13.581965 7960 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32a4954b64f244c6aa3aee7bdcf49528: Neither blocks nor log segments found. Creating new log.
I20260924 04:14:13.582839 7960 log.cc:824] T 00000000000000000000000000000000 P 32a4954b64f244c6aa3aee7bdcf49528: Log is configured to *not* fsync() on all Append() calls
I20260924 04:14:13.584851 7960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32a4954b64f244c6aa3aee7bdcf49528: No bootstrap required, opened a new log
Sep 24 04:14:13 dist-test-slave-p401 krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:14:13.588914 7960 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32a4954b64f244c6aa3aee7bdcf49528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32a4954b64f244c6aa3aee7bdcf49528" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39479 } }
I20260924 04:14:13.589128 7960 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32a4954b64f244c6aa3aee7bdcf49528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:14:13.589160 7960 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32a4954b64f244c6aa3aee7bdcf49528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32a4954b64f244c6aa3aee7bdcf49528, State: Initialized, Role: FOLLOWER
I20260924 04:14:13.589625 7960 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32a4954b64f244c6aa3aee7bdcf49528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32a4954b64f244c6aa3aee7bdcf49528" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39479 } }
I20260924 04:14:13.589792 7960 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32a4954b64f244c6aa3aee7bdcf49528 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:14:13.589874 7960 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32a4954b64f244c6aa3aee7bdcf49528 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:14:13.589967 7960 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32a4954b64f244c6aa3aee7bdcf49528 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:14:13.590772 7960 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32a4954b64f244c6aa3aee7bdcf49528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32a4954b64f244c6aa3aee7bdcf49528" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39479 } }
I20260924 04:14:13.591125 7960 leader_election.cc:302] T 00000000000000000000000000000000 P 32a4954b64f244c6aa3aee7bdcf49528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32a4954b64f244c6aa3aee7bdcf49528; no voters:
I20260924 04:14:13.591495 7960 leader_election.cc:288] T 00000000000000000000000000000000 P 32a4954b64f244c6aa3aee7bdcf49528 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:14:13.591559 7965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32a4954b64f244c6aa3aee7bdcf49528 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:14:13.591735 7965 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32a4954b64f244c6aa3aee7bdcf49528 [term 1 LEADER]: Becoming Leader. State: Replica: 32a4954b64f244c6aa3aee7bdcf49528, State: Running, Role: LEADER
I20260924 04:14:13.592020 7965 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32a4954b64f244c6aa3aee7bdcf49528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32a4954b64f244c6aa3aee7bdcf49528" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39479 } }
I20260924 04:14:13.592234 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:13.579576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59688 (local address 127.0.98.62:39479)
0924 04:14:13.579977 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:13.579994 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:13.580034 (+ 40us) server_negotiation.cc:407] Connection header received
0924 04:14:13.580844 (+ 810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:13.580866 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:13.581264 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:13.581625 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:13.582266 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:13.583602 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:13.584409 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:13.585055 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:13.585210 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:13.587723 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:13.587728 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:13.587740 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:13.587764 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:13.589295 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:13.589750 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:13.589755 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:13.589761 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:13.589893 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:13.590151 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:13.590154 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:13.590156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:13.590537 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:13.590713 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:13.590969 (+ 256us) server_negotiation.cc:299] Negotiation successful
0924 04:14:13.591258 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":107,"threads_started":1}
I20260924 04:14:13.592725 7960 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32a4954b64f244c6aa3aee7bdcf49528 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:14:13.593508 7967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32a4954b64f244c6aa3aee7bdcf49528 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32a4954b64f244c6aa3aee7bdcf49528" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32a4954b64f244c6aa3aee7bdcf49528" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39479 } } }
I20260924 04:14:13.593621 7967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32a4954b64f244c6aa3aee7bdcf49528 [sys.catalog]: This master's current role is: LEADER
I20260924 04:14:13.593938 7966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32a4954b64f244c6aa3aee7bdcf49528 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32a4954b64f244c6aa3aee7bdcf49528. Latest consensus state: current_term: 1 leader_uuid: "32a4954b64f244c6aa3aee7bdcf49528" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32a4954b64f244c6aa3aee7bdcf49528" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39479 } } }
I20260924 04:14:13.594009 7966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32a4954b64f244c6aa3aee7bdcf49528 [sys.catalog]: This master's current role is: LEADER
I20260924 04:14:13.594254 7974 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:14:13.597291 7974 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:14:13.602579 7974 catalog_manager.cc:1347] Generated new cluster ID: bf58b47b613d436b9d561699ba8725e9
I20260924 04:14:13.602661 7974 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:14:13.616348 7974 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:14:13.617614 7974 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:14:13.634577 7974 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32a4954b64f244c6aa3aee7bdcf49528: Generated new TSK 0
I20260924 04:14:13.635440 7974 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260924 04:14:13.659974 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39479
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:14:13.767025 7986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:14:13.767315 7986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:14:13.767364 7986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:14:13.771514 7986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:14:13.771574 7986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:14:13.771612 7986 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 04:14:13.771664 7986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:14:13.776938 7986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:14:13.777999 7986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:14:13.779027 7986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:14:13.788542 7991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:13.788606 7992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:13.788631 7986 server_base.cc:1034] running on GCE node
W20260924 04:14:13.788542 7994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:13.789188 7986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:14:13.789808 7986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:14:13.790963 7986 hybrid_clock.cc:648] HybridClock initialized: now 1790223253790943 us; error 29 us; skew 500 ppm
Sep 24 04:14:13 dist-test-slave-p401 krb5kdc[7869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223253, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:14:13.793581 7986 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 04:14:13.794435 7986 webserver.cc:460] Webserver started at http://127.0.98.1:44759/ using document root <none> and password file <none>
I20260924 04:14:13.795007 7986 fs_manager.cc:359] Metadata directory not provided
I20260924 04:14:13.795068 7986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:14:13.795303 7986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:14:13.797281 7986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/ts-0/data/instance:
uuid: "c4fdab40943144c38ed46b59ce32516a"
format_stamp: "Formatted at 2026-09-24 04:14:13 on dist-test-slave-p401"
server_key: "98482635c42f9ac6a5cc0c117d5dd5f1"
server_key_iv: "ed070fba9757f580e952a1c63f9160fd"
server_key_version: "encryptionkey@0"
I20260924 04:14:13.797812 7986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c4fdab40943144c38ed46b59ce32516a"
format_stamp: "Formatted at 2026-09-24 04:14:13 on dist-test-slave-p401"
server_key: "98482635c42f9ac6a5cc0c117d5dd5f1"
server_key_iv: "ed070fba9757f580e952a1c63f9160fd"
server_key_version: "encryptionkey@0"
I20260924 04:14:13.801376 7986 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:14:13.803779 8001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:13.805008 7986 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:14:13.805135 7986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/ts-0/wal
uuid: "c4fdab40943144c38ed46b59ce32516a"
format_stamp: "Formatted at 2026-09-24 04:14:13 on dist-test-slave-p401"
server_key: "98482635c42f9ac6a5cc0c117d5dd5f1"
server_key_iv: "ed070fba9757f580e952a1c63f9160fd"
server_key_version: "encryptionkey@0"
I20260924 04:14:13.805239 7986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:14:13.816313 7986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:14:13.819013 7986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:14:13.819196 7986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:14:13.820694 7986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:14:13.820746 7986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:13.820799 7986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:14:13.820834 7986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 04:14:13 dist-test-slave-p401 krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223253, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:14:13.831744 7986 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42829
I20260924 04:14:13.832818 7986 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:14:13.834302 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:13.821275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41627 (local address 127.0.98.62:39479)
0924 04:14:13.821380 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:13.821385 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:13.822555 (+ 1170us) server_negotiation.cc:407] Connection header received
0924 04:14:13.823566 (+ 1011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:13.823569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:13.823626 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:13.823696 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:13.825003 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:13.825551 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:13.826797 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:13.827033 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:13.827120 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:13.830548 (+ 3428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:13.830551 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:13.830553 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:13.830576 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:13.832203 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:13.832668 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:13.832673 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:13.832675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:13.832778 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:13.833137 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:13.833141 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:13.833143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:13.833388 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:13.833497 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:13.834089 (+ 592us) server_negotiation.cc:299] Negotiation successful
0924 04:14:13.834189 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 04:14:13.833550 8116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42829 every 8 connection(s)
I20260924 04:14:13.835052 8010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:13.821826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39479 (local address 127.0.98.1:41627)
0924 04:14:13.822399 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:13.822441 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:13.823360 (+ 919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:13.823972 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:13.823981 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:13.824373 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:13.824858 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:13.824872 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:13.825704 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:13.825707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:13.826686 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:13.826696 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:13.827167 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:13.827171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:13.827364 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:13.827932 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:13.827955 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:13.830388 (+ 2433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:14:13.832315 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:13.832319 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:13.832337 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:13.832566 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:13.832874 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:13.832877 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:13.832879 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:13.833024 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:13.833554 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:14:13.833560 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:14:13.833852 (+ 292us) client_negotiation.cc:769] Sending connection context
0924 04:14:13.834026 (+ 174us) client_negotiation.cc:240] Negotiation successful
0924 04:14:13.834279 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":200,"threads_started":1}
I20260924 04:14:13.835687 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 7986
I20260924 04:14:13.835809 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:14:13.836133 392 external_mini_cluster.cc:1428] Setting key b2620c1fee05b0ec8fe6263b5777ffdb
I20260924 04:14:13.836651 8117 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39479
I20260924 04:14:13.836789 8117 heartbeater.cc:464] Registering TS with master...
I20260924 04:14:13.837289 8117 heartbeater.cc:511] Master 127.0.98.62:39479 requested a full tablet report, sending...
I20260924 04:14:13.840137 7910 ts_manager.cc:194] Registered new tserver with Master: c4fdab40943144c38ed46b59ce32516a (127.0.98.1:42829)
I20260924 04:14:13.841252 7910 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41627
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260924 04:14:13.848397 8123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:13.838503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39479 (local address 127.0.98.1:33513)
0924 04:14:13.839410 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:13.839425 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:13.839520 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:13.841983 (+ 2463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:13.841986 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:13.842200 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:13.842328 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:13.842336 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:13.843382 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:13.843385 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:13.844031 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:13.844040 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:13.844473 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:13.844477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:13.844575 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:13.844997 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:13.845009 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:13.845352 (+ 343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:14:13.846898 (+ 1546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:13.846900 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:13.846902 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:13.847107 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:13.847398 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:13.847400 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:13.847402 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:13.847475 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:13.847881 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:14:13.847884 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:14:13.847973 (+ 89us) client_negotiation.cc:769] Sending connection context
0924 04:14:13.848067 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 04:14:13.848199 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":820,"thread_start_us":105,"threads_started":1}
I20260924 04:14:13.848404 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:13.838559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33513 (local address 127.0.98.62:39479)
0924 04:14:13.838654 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:13.838657 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:13.841767 (+ 3110us) server_negotiation.cc:407] Connection header received
0924 04:14:13.841802 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:13.841803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:13.841846 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:13.841931 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:13.842456 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:13.843190 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:13.844150 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:13.844360 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:13.844449 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:13.845489 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:13.845493 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:13.845495 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:13.845517 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:13.846782 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:13.847198 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:13.847202 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:13.847203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:13.847318 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:13.847547 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:13.847550 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:13.847551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:13.847743 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:13.847837 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:13.848137 (+ 300us) server_negotiation.cc:299] Negotiation successful
0924 04:14:13.848250 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260924 04:14:13.862368 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39479
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:14:13.966790 8126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:14:13.967056 8126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:14:13.967113 8126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:14:13.971093 8126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:14:13.971159 8126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:14:13.971211 8126 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 04:14:13.971282 8126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:14:13.976514 8126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:14:13.977517 8126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:14:13.978515 8126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:14:13.987810 8126 server_base.cc:1034] running on GCE node
W20260924 04:14:13.987763 8131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:13.987758 8134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:13.987763 8132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:13.988364 8126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:14:13.988956 8126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:14:13.990106 8126 hybrid_clock.cc:648] HybridClock initialized: now 1790223253990087 us; error 32 us; skew 500 ppm
Sep 24 04:14:13 dist-test-slave-p401 krb5kdc[7869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223253, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:14:13.996795 8126 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 04:14:13.997702 8126 webserver.cc:460] Webserver started at http://127.0.98.2:46733/ using document root <none> and password file <none>
I20260924 04:14:13.998222 8126 fs_manager.cc:359] Metadata directory not provided
I20260924 04:14:13.998286 8126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:14:13.998482 8126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:14:14.000501 8126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/ts-1/data/instance:
uuid: "a22832de64344223ba8985121126520f"
format_stamp: "Formatted at 2026-09-24 04:14:13 on dist-test-slave-p401"
server_key: "73350314cefac7e4acb37a659ff9fdd5"
server_key_iv: "2a9834cfa84ab21ed36fa2d88f69f20b"
server_key_version: "encryptionkey@0"
I20260924 04:14:14.001024 8126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a22832de64344223ba8985121126520f"
format_stamp: "Formatted at 2026-09-24 04:14:13 on dist-test-slave-p401"
server_key: "73350314cefac7e4acb37a659ff9fdd5"
server_key_iv: "2a9834cfa84ab21ed36fa2d88f69f20b"
server_key_version: "encryptionkey@0"
I20260924 04:14:14.004601 8126 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:14:14.006870 8141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:14.008059 8126 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:14:14.008162 8126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/ts-1/wal
uuid: "a22832de64344223ba8985121126520f"
format_stamp: "Formatted at 2026-09-24 04:14:13 on dist-test-slave-p401"
server_key: "73350314cefac7e4acb37a659ff9fdd5"
server_key_iv: "2a9834cfa84ab21ed36fa2d88f69f20b"
server_key_version: "encryptionkey@0"
I20260924 04:14:14.008257 8126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:14:14.019812 8126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:14:14.022441 8126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:14:14.022607 8126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:14:14.024027 8126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:14:14.024077 8126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:14.024129 8126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:14:14.024158 8126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 04:14:14 dist-test-slave-p401 krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223253, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:14:14.034037 8126 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44637
I20260924 04:14:14.034271 8256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44637 every 8 connection(s)
I20260924 04:14:14.034936 8126 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:14:14.036698 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:14.024662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36411 (local address 127.0.98.62:39479)
0924 04:14:14.024807 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:14.024811 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:14.025805 (+ 994us) server_negotiation.cc:407] Connection header received
0924 04:14:14.027122 (+ 1317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:14.027126 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:14.027182 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:14.027383 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:14.028562 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:14.029141 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:14.030234 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:14.030592 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:14.030695 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:14.033165 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:14.033169 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:14.033172 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:14.033198 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:14.034675 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:14.035207 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:14.035211 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:14.035213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:14.035337 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:14.035699 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:14.035702 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:14.035704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:14.035914 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:14.035989 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:14.036432 (+ 443us) server_negotiation.cc:299] Negotiation successful
0924 04:14:14.036528 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 04:14:14.037431 8150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:14.025048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39479 (local address 127.0.98.2:36411)
0924 04:14:14.025605 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:14.025690 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:14.026870 (+ 1180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:14.027546 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:14.027553 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:14.027874 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:14.028424 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:14.028440 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:14.029246 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:14.029249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:14.030120 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:14.030129 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:14.030749 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:14.030753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:14.030915 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:14.031502 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:14.031517 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:14.033012 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:14:14.034777 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:14.034783 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:14.034797 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:14.035118 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:14.035442 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:14.035445 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:14.035447 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:14.035598 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:14.036003 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:14:14.036008 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:14:14.036225 (+ 217us) client_negotiation.cc:769] Sending connection context
0924 04:14:14.036380 (+ 155us) client_negotiation.cc:240] Negotiation successful
0924 04:14:14.036665 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":108,"threads_started":1}
I20260924 04:14:14.037629 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 8126
I20260924 04:14:14.037757 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:14:14.038056 392 external_mini_cluster.cc:1428] Setting key 591f293ee4d0edce8699504fb5d3d7ff
I20260924 04:14:14.038630 8257 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39479
I20260924 04:14:14.038846 8257 heartbeater.cc:464] Registering TS with master...
I20260924 04:14:14.039415 8257 heartbeater.cc:511] Master 127.0.98.62:39479 requested a full tablet report, sending...
I20260924 04:14:14.041633 7910 ts_manager.cc:194] Registered new tserver with Master: a22832de64344223ba8985121126520f (127.0.98.2:44637)
I20260924 04:14:14.042174 7910 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36411
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:14:14.048333 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:14.042479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57809 (local address 127.0.98.62:39479)
0924 04:14:14.042581 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:14.042584 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:14.042594 (+ 10us) server_negotiation.cc:407] Connection header received
0924 04:14:14.042619 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:14.042621 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:14.042652 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:14.042714 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:14.043115 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:14.043660 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:14.044424 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:14.044626 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:14.044704 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:14.045646 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:14.045649 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:14.045652 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:14.045675 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:14.046892 (+ 1217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:14.047293 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:14.047297 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:14.047298 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:14.047385 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:14.047618 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:14.047620 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:14.047622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:14.047833 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:14.047908 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:14.048102 (+ 194us) server_negotiation.cc:299] Negotiation successful
0924 04:14:14.048205 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 04:14:14.048336 8264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:14.041454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39479 (local address 127.0.98.2:57809)
0924 04:14:14.041726 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:14.041741 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:14.041822 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:14.042752 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:14.042755 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:14.042930 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:14.043020 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:14.043026 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:14.043767 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:14.043770 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:14.044326 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:14.044335 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:14.044714 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:14.044718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:14.044802 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:14.045229 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:14.045240 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:14.045530 (+ 290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:14:14.046983 (+ 1453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:14.046986 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:14.046988 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:14.047162 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:14.047472 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:14.047474 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:14.047475 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:14.047542 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:14.047909 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:14:14.047911 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:14:14.047981 (+ 70us) client_negotiation.cc:769] Sending connection context
0924 04:14:14.048057 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 04:14:14.048156 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":98,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260924 04:14:14.063493 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39479
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:14:14.183022 8266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:14:14.183318 8266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:14:14.183377 8266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:14:14.187683 8266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:14:14.187739 8266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:14:14.187783 8266 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 04:14:14.187835 8266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:14:14.193027 8266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:14:14.194021 8266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:14:14.194998 8266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:14:14.204077 8272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:14.204094 8274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:14.204082 8271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:14.204344 8266 server_base.cc:1034] running on GCE node
I20260924 04:14:14.204790 8266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:14:14.205399 8266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:14:14.206553 8266 hybrid_clock.cc:648] HybridClock initialized: now 1790223254206536 us; error 30 us; skew 500 ppm
Sep 24 04:14:14 dist-test-slave-p401 krb5kdc[7869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223254, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:14:14.209153 8266 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 04:14:14.209985 8266 webserver.cc:460] Webserver started at http://127.0.98.3:38857/ using document root <none> and password file <none>
I20260924 04:14:14.210480 8266 fs_manager.cc:359] Metadata directory not provided
I20260924 04:14:14.210533 8266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:14:14.210687 8266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:14:14.212769 8266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/ts-2/data/instance:
uuid: "13fd4d44db624936bdedc092671ec745"
format_stamp: "Formatted at 2026-09-24 04:14:14 on dist-test-slave-p401"
server_key: "aaf1b70f22c65eca074f35730c7d1a42"
server_key_iv: "57e13901e4b02223513aa20f5195557a"
server_key_version: "encryptionkey@0"
I20260924 04:14:14.213271 8266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/ts-2/wal/instance:
uuid: "13fd4d44db624936bdedc092671ec745"
format_stamp: "Formatted at 2026-09-24 04:14:14 on dist-test-slave-p401"
server_key: "aaf1b70f22c65eca074f35730c7d1a42"
server_key_iv: "57e13901e4b02223513aa20f5195557a"
server_key_version: "encryptionkey@0"
I20260924 04:14:14.216701 8266 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:14:14.218962 8281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:14.220132 8266 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:14:14.220223 8266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/ts-2/wal
uuid: "13fd4d44db624936bdedc092671ec745"
format_stamp: "Formatted at 2026-09-24 04:14:14 on dist-test-slave-p401"
server_key: "aaf1b70f22c65eca074f35730c7d1a42"
server_key_iv: "57e13901e4b02223513aa20f5195557a"
server_key_version: "encryptionkey@0"
I20260924 04:14:14.220302 8266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:14:14.233942 8266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:14:14.236629 8266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:14:14.236780 8266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:14:14.238267 8266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:14:14.238322 8266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:14.238376 8266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:14:14.238407 8266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 04:14:14 dist-test-slave-p401 krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223254, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:14:14.248179 8266 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36673
I20260924 04:14:14.248247 8396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36673 every 8 connection(s)
I20260924 04:14:14.249034 8266 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:14:14.250861 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:14.238881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54167 (local address 127.0.98.62:39479)
0924 04:14:14.239016 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:14.239021 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:14.240085 (+ 1064us) server_negotiation.cc:407] Connection header received
0924 04:14:14.241087 (+ 1002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:14.241092 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:14.241147 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:14.241199 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:14.242795 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:14.243534 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:14.244683 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:14.244902 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:14.244989 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:14.247392 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:14.247396 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:14.247399 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:14.247426 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:14.248922 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:14.249433 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:14.249436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:14.249438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:14.249525 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:14.249882 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:14.249886 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:14.249888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:14.250119 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:14.250208 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:14.250639 (+ 431us) server_negotiation.cc:299] Negotiation successful
0924 04:14:14.250739 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 04:14:14.251626 8290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:14.239460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39479 (local address 127.0.98.3:54167)
0924 04:14:14.239934 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:14.239975 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:14.240867 (+ 892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:14.241591 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:14.241600 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:14.242027 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:14.242656 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:14.242674 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:14.243649 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:14.243652 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:14.244573 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:14.244582 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:14.245046 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:14.245050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:14.245204 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:14.245721 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:14.245736 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:14.247211 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:14:14.249019 (+ 1808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:14.249025 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:14.249040 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:14.249316 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:14.249621 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:14.249624 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:14.249626 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:14.249781 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:14.250211 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:14:14.250215 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:14:14.250430 (+ 215us) client_negotiation.cc:769] Sending connection context
0924 04:14:14.250575 (+ 145us) client_negotiation.cc:240] Negotiation successful
0924 04:14:14.250807 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":118,"threads_started":1}
I20260924 04:14:14.252810 8397 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39479
I20260924 04:14:14.252940 8397 heartbeater.cc:464] Registering TS with master...
I20260924 04:14:14.253419 8397 heartbeater.cc:511] Master 127.0.98.62:39479 requested a full tablet report, sending...
I20260924 04:14:14.254359 7914 ts_manager.cc:194] Registered new tserver with Master: 13fd4d44db624936bdedc092671ec745 (127.0.98.3:36673)
I20260924 04:14:14.255085 7914 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54167
I20260924 04:14:14.259119 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 8266
I20260924 04:14:14.259207 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:14:14.259464 392 external_mini_cluster.cc:1428] Setting key 80db9d2508ec74e02d651f5926573068
I20260924 04:14:14.260674 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:14.254683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44279 (local address 127.0.98.62:39479)
0924 04:14:14.254787 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:14.254790 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:14.254899 (+ 109us) server_negotiation.cc:407] Connection header received
0924 04:14:14.254986 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:14.254989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:14.255029 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:14.255097 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:14.255509 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:14.256184 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:14.256970 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:14.257174 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:14.257249 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:14.258130 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:14.258132 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:14.258134 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:14.258148 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:14.259190 (+ 1042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:14.259630 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:14.259633 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:14.259635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:14.259716 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:14.259950 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:14.259953 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:14.259955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:14.260115 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:14.260196 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:14.260437 (+ 241us) server_negotiation.cc:299] Negotiation successful
0924 04:14:14.260531 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 04:14:14.260684 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:14.254600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39479 (local address 127.0.98.3:44279)
0924 04:14:14.254826 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:14.254839 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:14.254912 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:14.255107 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:14.255109 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:14.255312 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:14.255425 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:14.255432 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:14.256313 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:14.256316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:14.256869 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:14.256876 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:14.257253 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:14.257255 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:14.257316 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:14.257741 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:14.257752 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:14.258038 (+ 286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:14:14.259343 (+ 1305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:14.259346 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:14.259347 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:14.259534 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:14.259793 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:14.259796 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:14.259798 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:14.259871 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:14.260198 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:14:14.260201 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:14:14.260282 (+ 81us) client_negotiation.cc:769] Sending connection context
0924 04:14:14.260372 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 04:14:14.260506 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":76,"threads_started":1}
I20260924 04:14:14.261298 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 04:14:14 dist-test-slave-p401 krb5kdc[7869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223254, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 04:14:14 dist-test-slave-p401 krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:14:14 dist-test-slave-p401 krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 04:14:14.283517 7963 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:14.277636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42288 (local address 127.0.98.62:39479)
0924 04:14:14.277794 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:14.277798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:14.277813 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:14:14.277865 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:14.277867 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:14.277908 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:14.278023 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:14.278751 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:14.279330 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:14.280141 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:14.280351 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:14.280438 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:14.283371 (+ 2933us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42288: BlockingRecv error: recv got EOF from 127.0.0.1:42288 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 24 04:14:14 dist-test-slave-p401 krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223254, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:14:14.291487 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:14.284750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42304 (local address 127.0.98.62:39479)
0924 04:14:14.284855 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:14.284858 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:14.284978 (+ 120us) server_negotiation.cc:407] Connection header received
0924 04:14:14.285057 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:14.285059 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:14.285093 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:14.285155 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:14.285644 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:14.286196 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:14.286920 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:14.287111 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:14.287178 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:14.289218 (+ 2040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:14.289221 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:14.289223 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:14.289243 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:14.290295 (+ 1052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:14.290626 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:14.290628 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:14.290630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:14.290689 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:14.290877 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:14.290879 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:14.290880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:14.291022 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:14.291108 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:14.291282 (+ 174us) server_negotiation.cc:299] Negotiation successful
0924 04:14:14.291375 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 04:14:14.294031 7910 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42304:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:14:14.297037 7910 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:14:14.311146 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:14.305058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56870 (local address 127.0.98.2:44637)
0924 04:14:14.305382 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:14.305395 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:14.305855 (+ 460us) server_negotiation.cc:407] Connection header received
0924 04:14:14.305938 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:14.305943 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:14.306160 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:14.306326 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:14:14.306769 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:14.307988 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:14.309634 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:14.310483 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:14.310585 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:14.310735 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:14.310875 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 04:14:14.310959 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":104,"threads_started":1}
I20260924 04:14:14.311136 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:14.304883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44637 (local address 127.0.0.1:56870)
0924 04:14:14.305529 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:14.305579 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:14.305711 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:14.306493 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:14.306498 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:14.306519 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:14:14.306651 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:14.306658 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:14.308116 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:14.308119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:14.309527 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:14.309535 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:14.310656 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:14.310659 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:14.310744 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:14.310761 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 04:14:14.310843 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 04:14:14.311002 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"mutex_wait_us":201,"thread_start_us":69,"threads_started":1}
I20260924 04:14:14.311141 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:14.305129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42829 (local address 127.0.0.1:53036)
0924 04:14:14.306152 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:14.306164 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:14.306254 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:14.306710 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:14.306713 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:14.306729 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:14:14.306838 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:14.306845 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:14.308128 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:14.308131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:14.309609 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:14.309619 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:14.310552 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:14.310555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:14.310694 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:14.310735 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 04:14:14.310852 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 04:14:14.310984 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":963,"mutex_wait_us":230,"thread_start_us":94,"threads_started":1}
I20260924 04:14:14.311583 8422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:14.305574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36673 (local address 127.0.0.1:38136)
0924 04:14:14.306248 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:14.306261 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:14.306352 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:14.306963 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:14.306966 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:14.306980 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:14:14.307039 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:14.307042 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:14.308082 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:14.308086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:14.309609 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:14.309619 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:14.310588 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:14.310591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:14.311014 (+ 423us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:14.311032 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 04:14:14.311095 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 04:14:14.311148 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"mutex_wait_us":439,"thread_start_us":82,"threads_started":1}
I20260924 04:14:14.311852 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:14.305734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38136 (local address 127.0.98.3:36673)
0924 04:14:14.306123 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:14.306134 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:14.306444 (+ 310us) server_negotiation.cc:407] Connection header received
0924 04:14:14.306520 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:14.306525 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:14.306711 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:14.306843 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:14:14.307131 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:14.307983 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:14.309725 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:14.310477 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:14.310585 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:14.310735 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:14.311217 (+ 482us) server_negotiation.cc:299] Negotiation successful
0924 04:14:14.311615 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":107,"threads_started":1}
I20260924 04:14:14.311903 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:14.305670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53036 (local address 127.0.98.1:42829)
0924 04:14:14.305939 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:14.305952 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:14.306335 (+ 383us) server_negotiation.cc:407] Connection header received
0924 04:14:14.306410 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:14.306415 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:14.306590 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:14.306736 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:14:14.306943 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:14.308052 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:14.309761 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:14.310295 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:14.310358 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:14.310476 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:14.311541 (+ 1065us) server_negotiation.cc:299] Negotiation successful
0924 04:14:14.311773 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":77,"threads_started":1}
I20260924 04:14:14.314189 8331 tablet_service.cc:1467] Processing CreateTablet for tablet 29f01da9f7654ae28a6d39f0f9212729 (DEFAULT_TABLE table=test-table [id=caa76cb035464a6da3684fab51d7abb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:14:14.314170 8191 tablet_service.cc:1467] Processing CreateTablet for tablet 29f01da9f7654ae28a6d39f0f9212729 (DEFAULT_TABLE table=test-table [id=caa76cb035464a6da3684fab51d7abb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:14:14.315049 8191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29f01da9f7654ae28a6d39f0f9212729. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:14:14.315058 8331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29f01da9f7654ae28a6d39f0f9212729. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:14:14.317174 8051 tablet_service.cc:1467] Processing CreateTablet for tablet 29f01da9f7654ae28a6d39f0f9212729 (DEFAULT_TABLE table=test-table [id=caa76cb035464a6da3684fab51d7abb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:14:14.318222 8051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29f01da9f7654ae28a6d39f0f9212729. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:14:14.321575 8425 tablet_bootstrap.cc:492] T 29f01da9f7654ae28a6d39f0f9212729 P 13fd4d44db624936bdedc092671ec745: Bootstrap starting.
I20260924 04:14:14.323158 8426 tablet_bootstrap.cc:492] T 29f01da9f7654ae28a6d39f0f9212729 P a22832de64344223ba8985121126520f: Bootstrap starting.
I20260924 04:14:14.324047 8425 tablet_bootstrap.cc:654] T 29f01da9f7654ae28a6d39f0f9212729 P 13fd4d44db624936bdedc092671ec745: Neither blocks nor log segments found. Creating new log.
I20260924 04:14:14.324828 8425 log.cc:824] T 29f01da9f7654ae28a6d39f0f9212729 P 13fd4d44db624936bdedc092671ec745: Log is configured to *not* fsync() on all Append() calls
I20260924 04:14:14.325254 8426 tablet_bootstrap.cc:654] T 29f01da9f7654ae28a6d39f0f9212729 P a22832de64344223ba8985121126520f: Neither blocks nor log segments found. Creating new log.
I20260924 04:14:14.325664 8428 tablet_bootstrap.cc:492] T 29f01da9f7654ae28a6d39f0f9212729 P c4fdab40943144c38ed46b59ce32516a: Bootstrap starting.
I20260924 04:14:14.326045 8426 log.cc:824] T 29f01da9f7654ae28a6d39f0f9212729 P a22832de64344223ba8985121126520f: Log is configured to *not* fsync() on all Append() calls
I20260924 04:14:14.326838 8425 tablet_bootstrap.cc:492] T 29f01da9f7654ae28a6d39f0f9212729 P 13fd4d44db624936bdedc092671ec745: No bootstrap required, opened a new log
I20260924 04:14:14.327023 8425 ts_tablet_manager.cc:1397] T 29f01da9f7654ae28a6d39f0f9212729 P 13fd4d44db624936bdedc092671ec745: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:14:14.327715 8426 tablet_bootstrap.cc:492] T 29f01da9f7654ae28a6d39f0f9212729 P a22832de64344223ba8985121126520f: No bootstrap required, opened a new log
I20260924 04:14:14.327824 8428 tablet_bootstrap.cc:654] T 29f01da9f7654ae28a6d39f0f9212729 P c4fdab40943144c38ed46b59ce32516a: Neither blocks nor log segments found. Creating new log.
I20260924 04:14:14.327903 8426 ts_tablet_manager.cc:1397] T 29f01da9f7654ae28a6d39f0f9212729 P a22832de64344223ba8985121126520f: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260924 04:14:14.328961 8428 log.cc:824] T 29f01da9f7654ae28a6d39f0f9212729 P c4fdab40943144c38ed46b59ce32516a: Log is configured to *not* fsync() on all Append() calls
I20260924 04:14:14.330885 8428 tablet_bootstrap.cc:492] T 29f01da9f7654ae28a6d39f0f9212729 P c4fdab40943144c38ed46b59ce32516a: No bootstrap required, opened a new log
I20260924 04:14:14.330776 8425 raft_consensus.cc:348] T 29f01da9f7654ae28a6d39f0f9212729 P 13fd4d44db624936bdedc092671ec745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 } } peers { permanent_uuid: "c4fdab40943144c38ed46b59ce32516a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } } peers { permanent_uuid: "13fd4d44db624936bdedc092671ec745" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36673 } }
I20260924 04:14:14.331039 8425 raft_consensus.cc:374] T 29f01da9f7654ae28a6d39f0f9212729 P 13fd4d44db624936bdedc092671ec745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:14:14.331064 8428 ts_tablet_manager.cc:1397] T 29f01da9f7654ae28a6d39f0f9212729 P c4fdab40943144c38ed46b59ce32516a: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260924 04:14:14.330909 8426 raft_consensus.cc:348] T 29f01da9f7654ae28a6d39f0f9212729 P a22832de64344223ba8985121126520f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 } } peers { permanent_uuid: "c4fdab40943144c38ed46b59ce32516a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } } peers { permanent_uuid: "13fd4d44db624936bdedc092671ec745" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36673 } }
I20260924 04:14:14.331094 8425 raft_consensus.cc:729] T 29f01da9f7654ae28a6d39f0f9212729 P 13fd4d44db624936bdedc092671ec745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13fd4d44db624936bdedc092671ec745, State: Initialized, Role: FOLLOWER
I20260924 04:14:14.331144 8426 raft_consensus.cc:374] T 29f01da9f7654ae28a6d39f0f9212729 P a22832de64344223ba8985121126520f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:14:14.331207 8426 raft_consensus.cc:729] T 29f01da9f7654ae28a6d39f0f9212729 P a22832de64344223ba8985121126520f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a22832de64344223ba8985121126520f, State: Initialized, Role: FOLLOWER
I20260924 04:14:14.331610 8426 consensus_queue.cc:260] T 29f01da9f7654ae28a6d39f0f9212729 P a22832de64344223ba8985121126520f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 } } peers { permanent_uuid: "c4fdab40943144c38ed46b59ce32516a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } } peers { permanent_uuid: "13fd4d44db624936bdedc092671ec745" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36673 } }
I20260924 04:14:14.331596 8425 consensus_queue.cc:260] T 29f01da9f7654ae28a6d39f0f9212729 P 13fd4d44db624936bdedc092671ec745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 } } peers { permanent_uuid: "c4fdab40943144c38ed46b59ce32516a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } } peers { permanent_uuid: "13fd4d44db624936bdedc092671ec745" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36673 } }
I20260924 04:14:14.332224 8257 heartbeater.cc:503] Master 127.0.98.62:39479 was elected leader, sending a full tablet report...
I20260924 04:14:14.332495 8426 ts_tablet_manager.cc:1428] T 29f01da9f7654ae28a6d39f0f9212729 P a22832de64344223ba8985121126520f: Time spent starting tablet: real 0.005s user 0.002s sys 0.002s
I20260924 04:14:14.332530 8425 ts_tablet_manager.cc:1428] T 29f01da9f7654ae28a6d39f0f9212729 P 13fd4d44db624936bdedc092671ec745: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260924 04:14:14.332818 8397 heartbeater.cc:503] Master 127.0.98.62:39479 was elected leader, sending a full tablet report...
I20260924 04:14:14.335495 8428 raft_consensus.cc:348] T 29f01da9f7654ae28a6d39f0f9212729 P c4fdab40943144c38ed46b59ce32516a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 } } peers { permanent_uuid: "c4fdab40943144c38ed46b59ce32516a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } } peers { permanent_uuid: "13fd4d44db624936bdedc092671ec745" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36673 } }
I20260924 04:14:14.335772 8428 raft_consensus.cc:374] T 29f01da9f7654ae28a6d39f0f9212729 P c4fdab40943144c38ed46b59ce32516a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:14:14.335826 8428 raft_consensus.cc:729] T 29f01da9f7654ae28a6d39f0f9212729 P c4fdab40943144c38ed46b59ce32516a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4fdab40943144c38ed46b59ce32516a, State: Initialized, Role: FOLLOWER
I20260924 04:14:14.336303 8428 consensus_queue.cc:260] T 29f01da9f7654ae28a6d39f0f9212729 P c4fdab40943144c38ed46b59ce32516a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 } } peers { permanent_uuid: "c4fdab40943144c38ed46b59ce32516a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } } peers { permanent_uuid: "13fd4d44db624936bdedc092671ec745" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36673 } }
I20260924 04:14:14.336961 8117 heartbeater.cc:503] Master 127.0.98.62:39479 was elected leader, sending a full tablet report...
I20260924 04:14:14.338124 8428 ts_tablet_manager.cc:1428] T 29f01da9f7654ae28a6d39f0f9212729 P c4fdab40943144c38ed46b59ce32516a: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260924 04:14:14.499969 8398 tablet.cc:2369] T 29f01da9f7654ae28a6d39f0f9212729 P 13fd4d44db624936bdedc092671ec745: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:14:14.517854 8433 raft_consensus.cc:482] T 29f01da9f7654ae28a6d39f0f9212729 P c4fdab40943144c38ed46b59ce32516a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:14:14.518064 8433 raft_consensus.cc:504] T 29f01da9f7654ae28a6d39f0f9212729 P c4fdab40943144c38ed46b59ce32516a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 } } peers { permanent_uuid: "c4fdab40943144c38ed46b59ce32516a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } } peers { permanent_uuid: "13fd4d44db624936bdedc092671ec745" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36673 } }
I20260924 04:14:14.519130 8433 leader_election.cc:288] T 29f01da9f7654ae28a6d39f0f9212729 P c4fdab40943144c38ed46b59ce32516a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a22832de64344223ba8985121126520f (127.0.98.2:44637), 13fd4d44db624936bdedc092671ec745 (127.0.98.3:36673)
I20260924 04:14:14.522753 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:14.519524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39223 (local address 127.0.98.3:36673)
0924 04:14:14.519651 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:14.519655 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:14.519941 (+ 286us) server_negotiation.cc:407] Connection header received
0924 04:14:14.520053 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:14.520056 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:14.520097 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:14.520155 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:14:14.520396 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:14.520885 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:14.521896 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:14.522265 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:14.522352 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:14.522414 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:14.522571 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 04:14:14.522633 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 04:14:14.522792 8435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:14.519413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36673 (local address 127.0.98.1:39223)
0924 04:14:14.519853 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:14.519864 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:14.519958 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:14.520173 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:14.520175 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:14.520190 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:14:14.520294 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:14.520299 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:14.520979 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:14.520982 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:14.521782 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:14.521790 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:14.522376 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:14.522380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:14.522467 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:14.522487 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 04:14:14.522547 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 04:14:14.522613 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"mutex_wait_us":158,"thread_start_us":88,"threads_started":1}
I20260924 04:14:14.522791 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:14.519513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46157 (local address 127.0.98.2:44637)
0924 04:14:14.519696 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:14.519701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:14.519793 (+ 92us) server_negotiation.cc:407] Connection header received
0924 04:14:14.519949 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:14.519951 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:14.519993 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:14.520050 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:14:14.520330 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:14.520847 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:14.521896 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:14.522282 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:14.522372 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:14.522430 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:14.522606 (+ 176us) server_negotiation.cc:299] Negotiation successful
0924 04:14:14.522671 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260924 04:14:14.522806 8434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:14.519413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44637 (local address 127.0.98.1:46157)
0924 04:14:14.519713 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:14.519726 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:14.519841 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:14.520075 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:14.520078 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:14.520093 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:14:14.520232 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:14.520238 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:14.520953 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:14.520957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:14.521780 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:14.521789 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:14.522395 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:14.522399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:14.522505 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:14.522525 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 04:14:14.522584 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 04:14:14.522653 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":21,"thread_start_us":84,"threads_started":1}
I20260924 04:14:14.523445 8351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29f01da9f7654ae28a6d39f0f9212729" candidate_uuid: "c4fdab40943144c38ed46b59ce32516a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13fd4d44db624936bdedc092671ec745" is_pre_election: true
I20260924 04:14:14.523445 8211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29f01da9f7654ae28a6d39f0f9212729" candidate_uuid: "c4fdab40943144c38ed46b59ce32516a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a22832de64344223ba8985121126520f" is_pre_election: true
I20260924 04:14:14.523758 8351 raft_consensus.cc:2454] T 29f01da9f7654ae28a6d39f0f9212729 P 13fd4d44db624936bdedc092671ec745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4fdab40943144c38ed46b59ce32516a in term 0.
I20260924 04:14:14.523758 8211 raft_consensus.cc:2454] T 29f01da9f7654ae28a6d39f0f9212729 P a22832de64344223ba8985121126520f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4fdab40943144c38ed46b59ce32516a in term 0.
I20260924 04:14:14.524257 8004 leader_election.cc:302] T 29f01da9f7654ae28a6d39f0f9212729 P c4fdab40943144c38ed46b59ce32516a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 13fd4d44db624936bdedc092671ec745, a22832de64344223ba8985121126520f, c4fdab40943144c38ed46b59ce32516a; no voters:
I20260924 04:14:14.524526 8433 raft_consensus.cc:2781] T 29f01da9f7654ae28a6d39f0f9212729 P c4fdab40943144c38ed46b59ce32516a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:14:14.524595 8433 raft_consensus.cc:482] T 29f01da9f7654ae28a6d39f0f9212729 P c4fdab40943144c38ed46b59ce32516a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:14:14.524675 8433 raft_consensus.cc:3037] T 29f01da9f7654ae28a6d39f0f9212729 P c4fdab40943144c38ed46b59ce32516a [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:14:14.525641 8433 raft_consensus.cc:504] T 29f01da9f7654ae28a6d39f0f9212729 P c4fdab40943144c38ed46b59ce32516a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 } } peers { permanent_uuid: "c4fdab40943144c38ed46b59ce32516a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } } peers { permanent_uuid: "13fd4d44db624936bdedc092671ec745" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36673 } }
I20260924 04:14:14.525957 8433 leader_election.cc:288] T 29f01da9f7654ae28a6d39f0f9212729 P c4fdab40943144c38ed46b59ce32516a [CANDIDATE]: Term 1 election: Requested vote from peers a22832de64344223ba8985121126520f (127.0.98.2:44637), 13fd4d44db624936bdedc092671ec745 (127.0.98.3:36673)
I20260924 04:14:14.526271 8351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29f01da9f7654ae28a6d39f0f9212729" candidate_uuid: "c4fdab40943144c38ed46b59ce32516a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13fd4d44db624936bdedc092671ec745"
I20260924 04:14:14.526284 8211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29f01da9f7654ae28a6d39f0f9212729" candidate_uuid: "c4fdab40943144c38ed46b59ce32516a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a22832de64344223ba8985121126520f"
I20260924 04:14:14.526376 8351 raft_consensus.cc:3037] T 29f01da9f7654ae28a6d39f0f9212729 P 13fd4d44db624936bdedc092671ec745 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:14:14.526403 8211 raft_consensus.cc:3037] T 29f01da9f7654ae28a6d39f0f9212729 P a22832de64344223ba8985121126520f [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:14:14.527462 8211 raft_consensus.cc:2454] T 29f01da9f7654ae28a6d39f0f9212729 P a22832de64344223ba8985121126520f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4fdab40943144c38ed46b59ce32516a in term 1.
I20260924 04:14:14.527693 8351 raft_consensus.cc:2454] T 29f01da9f7654ae28a6d39f0f9212729 P 13fd4d44db624936bdedc092671ec745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4fdab40943144c38ed46b59ce32516a in term 1.
I20260924 04:14:14.527753 8004 leader_election.cc:302] T 29f01da9f7654ae28a6d39f0f9212729 P c4fdab40943144c38ed46b59ce32516a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a22832de64344223ba8985121126520f, c4fdab40943144c38ed46b59ce32516a; no voters:
I20260924 04:14:14.527899 8433 raft_consensus.cc:2781] T 29f01da9f7654ae28a6d39f0f9212729 P c4fdab40943144c38ed46b59ce32516a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:14:14.528076 8433 raft_consensus.cc:686] T 29f01da9f7654ae28a6d39f0f9212729 P c4fdab40943144c38ed46b59ce32516a [term 1 LEADER]: Becoming Leader. State: Replica: c4fdab40943144c38ed46b59ce32516a, State: Running, Role: LEADER
I20260924 04:14:14.528375 8433 consensus_queue.cc:237] T 29f01da9f7654ae28a6d39f0f9212729 P c4fdab40943144c38ed46b59ce32516a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 } } peers { permanent_uuid: "c4fdab40943144c38ed46b59ce32516a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } } peers { permanent_uuid: "13fd4d44db624936bdedc092671ec745" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36673 } }
I20260924 04:14:14.531469 7914 catalog_manager.cc:5495] T 29f01da9f7654ae28a6d39f0f9212729 P c4fdab40943144c38ed46b59ce32516a reported cstate change: term changed from 0 to 1, leader changed from <none> to c4fdab40943144c38ed46b59ce32516a (127.0.98.1). New cstate: current_term: 1 leader_uuid: "c4fdab40943144c38ed46b59ce32516a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4fdab40943144c38ed46b59ce32516a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13fd4d44db624936bdedc092671ec745" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36673 } health_report { overall_health: UNKNOWN } } }
W20260924 04:14:14.535965 8258 tablet.cc:2369] T 29f01da9f7654ae28a6d39f0f9212729 P a22832de64344223ba8985121126520f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:14:14.546381 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:14.543499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53042 (local address 127.0.98.1:42829)
0924 04:14:14.543603 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:14.543606 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:14.543618 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:14:14.543731 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:14.543733 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:14.543785 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:14.543873 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:14:14.544037 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:14.544521 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:14.545252 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:14.545455 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:14.545519 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:14.545699 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:14:14.546102 (+ 403us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:14:14.546191 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:14.546223 (+ 32us) server_negotiation.cc:299] Negotiation successful
0924 04:14:14.546266 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 04:14:14.551824 8351 raft_consensus.cc:1261] T 29f01da9f7654ae28a6d39f0f9212729 P 13fd4d44db624936bdedc092671ec745 [term 1 FOLLOWER]: Refusing update from remote peer c4fdab40943144c38ed46b59ce32516a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:14:14.551836 8211 raft_consensus.cc:1261] T 29f01da9f7654ae28a6d39f0f9212729 P a22832de64344223ba8985121126520f [term 1 FOLLOWER]: Refusing update from remote peer c4fdab40943144c38ed46b59ce32516a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:14:14.552490 8433 consensus_queue.cc:1035] T 29f01da9f7654ae28a6d39f0f9212729 P c4fdab40943144c38ed46b59ce32516a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:14:14.552716 8436 consensus_queue.cc:1035] T 29f01da9f7654ae28a6d39f0f9212729 P c4fdab40943144c38ed46b59ce32516a [LEADER]: Connected to new peer: Peer: permanent_uuid: "13fd4d44db624936bdedc092671ec745" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:14:14.559973 8440 mvcc.cc:205] Tried to move back new op lower bound from 7332754450637877248 to 7332754450550771712. Current Snapshot: MvccSnapshot[applied={T|T < 7332754450637877248}]
I20260924 04:14:14.568068 7914 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:14:14.568248 7914 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:14:14.570654 7914 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 32a4954b64f244c6aa3aee7bdcf49528: Sending DeleteTablet for 3 replicas of tablet 29f01da9f7654ae28a6d39f0f9212729
I20260924 04:14:14.571439 8191 tablet_service.cc:1514] Processing DeleteTablet for tablet 29f01da9f7654ae28a6d39f0f9212729 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:14:14 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56870
I20260924 04:14:14.571918 8449 tablet_replica.cc:330] T 29f01da9f7654ae28a6d39f0f9212729 P a22832de64344223ba8985121126520f: stopping tablet replica
I20260924 04:14:14.572227 8449 raft_consensus.cc:2229] T 29f01da9f7654ae28a6d39f0f9212729 P a22832de64344223ba8985121126520f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:14:14.572511 8449 raft_consensus.cc:2258] T 29f01da9f7654ae28a6d39f0f9212729 P a22832de64344223ba8985121126520f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:14:14.573884 8449 ts_tablet_manager.cc:1905] T 29f01da9f7654ae28a6d39f0f9212729 P a22832de64344223ba8985121126520f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:14:14.575877 8331 tablet_service.cc:1514] Processing DeleteTablet for tablet 29f01da9f7654ae28a6d39f0f9212729 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:14:14 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38136
I20260924 04:14:14.576519 8449 ts_tablet_manager.cc:1918] T 29f01da9f7654ae28a6d39f0f9212729 P a22832de64344223ba8985121126520f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:14:14.576609 8449 log.cc:1196] T 29f01da9f7654ae28a6d39f0f9212729 P a22832de64344223ba8985121126520f: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/ts-1/wal/wals/29f01da9f7654ae28a6d39f0f9212729
I20260924 04:14:14.576750 8051 tablet_service.cc:1514] Processing DeleteTablet for tablet 29f01da9f7654ae28a6d39f0f9212729 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:14:14 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53036
I20260924 04:14:14.576867 8449 ts_tablet_manager.cc:1939] T 29f01da9f7654ae28a6d39f0f9212729 P a22832de64344223ba8985121126520f: Deleting consensus metadata
I20260924 04:14:14.577317 7914 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42304:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:14:14.577883 7914 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:14:14.577992 7900 catalog_manager.cc:4841] TS a22832de64344223ba8985121126520f (127.0.98.2:44637): tablet 29f01da9f7654ae28a6d39f0f9212729 (table test-table [id=caa76cb035464a6da3684fab51d7abb1]) successfully deleted
I20260924 04:14:14.578812 8451 tablet_replica.cc:330] T 29f01da9f7654ae28a6d39f0f9212729 P c4fdab40943144c38ed46b59ce32516a: stopping tablet replica
I20260924 04:14:14.579154 8451 raft_consensus.cc:2229] T 29f01da9f7654ae28a6d39f0f9212729 P c4fdab40943144c38ed46b59ce32516a [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:14:14.579577 8451 raft_consensus.cc:2258] T 29f01da9f7654ae28a6d39f0f9212729 P c4fdab40943144c38ed46b59ce32516a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:14:14.582486 8051 tablet_service.cc:1467] Processing CreateTablet for tablet 3dd7d74d54a4439c97eca0d341d736ce (DEFAULT_TABLE table=test-table [id=a822988f3c504eff8e8f5f68294be8e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:14:14.582605 8191 tablet_service.cc:1467] Processing CreateTablet for tablet 3dd7d74d54a4439c97eca0d341d736ce (DEFAULT_TABLE table=test-table [id=a822988f3c504eff8e8f5f68294be8e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:14:14.582736 8051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dd7d74d54a4439c97eca0d341d736ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:14:14.582839 8191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dd7d74d54a4439c97eca0d341d736ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:14:14.585165 8426 tablet_bootstrap.cc:492] T 3dd7d74d54a4439c97eca0d341d736ce P a22832de64344223ba8985121126520f: Bootstrap starting.
I20260924 04:14:14.586061 8426 tablet_bootstrap.cc:654] T 3dd7d74d54a4439c97eca0d341d736ce P a22832de64344223ba8985121126520f: Neither blocks nor log segments found. Creating new log.
I20260924 04:14:14.586160 8450 tablet_replica.cc:330] T 29f01da9f7654ae28a6d39f0f9212729 P 13fd4d44db624936bdedc092671ec745: stopping tablet replica
I20260924 04:14:14.586458 8450 raft_consensus.cc:2229] T 29f01da9f7654ae28a6d39f0f9212729 P 13fd4d44db624936bdedc092671ec745 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:14:14.586755 8450 raft_consensus.cc:2258] T 29f01da9f7654ae28a6d39f0f9212729 P 13fd4d44db624936bdedc092671ec745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:14:14.586977 8428 tablet_bootstrap.cc:492] T 3dd7d74d54a4439c97eca0d341d736ce P c4fdab40943144c38ed46b59ce32516a: Bootstrap starting.
I20260924 04:14:14.587880 8428 tablet_bootstrap.cc:654] T 3dd7d74d54a4439c97eca0d341d736ce P c4fdab40943144c38ed46b59ce32516a: Neither blocks nor log segments found. Creating new log.
I20260924 04:14:14.587976 8450 ts_tablet_manager.cc:1905] T 29f01da9f7654ae28a6d39f0f9212729 P 13fd4d44db624936bdedc092671ec745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:14:14.589408 8331 tablet_service.cc:1467] Processing CreateTablet for tablet 3dd7d74d54a4439c97eca0d341d736ce (DEFAULT_TABLE table=test-table [id=a822988f3c504eff8e8f5f68294be8e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:14:14.589638 8331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dd7d74d54a4439c97eca0d341d736ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:14:14.590092 8428 tablet_bootstrap.cc:492] T 3dd7d74d54a4439c97eca0d341d736ce P c4fdab40943144c38ed46b59ce32516a: No bootstrap required, opened a new log
I20260924 04:14:14.590168 8428 ts_tablet_manager.cc:1397] T 3dd7d74d54a4439c97eca0d341d736ce P c4fdab40943144c38ed46b59ce32516a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 04:14:14.590305 8450 ts_tablet_manager.cc:1918] T 29f01da9f7654ae28a6d39f0f9212729 P 13fd4d44db624936bdedc092671ec745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:14:14.590389 8450 log.cc:1196] T 29f01da9f7654ae28a6d39f0f9212729 P 13fd4d44db624936bdedc092671ec745: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/ts-2/wal/wals/29f01da9f7654ae28a6d39f0f9212729
I20260924 04:14:14.590595 8428 raft_consensus.cc:348] T 3dd7d74d54a4439c97eca0d341d736ce P c4fdab40943144c38ed46b59ce32516a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4fdab40943144c38ed46b59ce32516a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } } peers { permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 } } peers { permanent_uuid: "13fd4d44db624936bdedc092671ec745" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36673 } }
I20260924 04:14:14.590659 8450 ts_tablet_manager.cc:1939] T 29f01da9f7654ae28a6d39f0f9212729 P 13fd4d44db624936bdedc092671ec745: Deleting consensus metadata
I20260924 04:14:14.590695 8428 raft_consensus.cc:374] T 3dd7d74d54a4439c97eca0d341d736ce P c4fdab40943144c38ed46b59ce32516a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:14:14.590723 8428 raft_consensus.cc:729] T 3dd7d74d54a4439c97eca0d341d736ce P c4fdab40943144c38ed46b59ce32516a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4fdab40943144c38ed46b59ce32516a, State: Initialized, Role: FOLLOWER
I20260924 04:14:14.590835 8428 consensus_queue.cc:260] T 3dd7d74d54a4439c97eca0d341d736ce P c4fdab40943144c38ed46b59ce32516a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4fdab40943144c38ed46b59ce32516a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } } peers { permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 } } peers { permanent_uuid: "13fd4d44db624936bdedc092671ec745" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36673 } }
I20260924 04:14:14.591073 8428 ts_tablet_manager.cc:1428] T 3dd7d74d54a4439c97eca0d341d736ce P c4fdab40943144c38ed46b59ce32516a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:14:14.591665 8425 tablet_bootstrap.cc:492] T 3dd7d74d54a4439c97eca0d341d736ce P 13fd4d44db624936bdedc092671ec745: Bootstrap starting.
I20260924 04:14:14.592082 8426 tablet_bootstrap.cc:492] T 3dd7d74d54a4439c97eca0d341d736ce P a22832de64344223ba8985121126520f: No bootstrap required, opened a new log
I20260924 04:14:14.592154 8426 ts_tablet_manager.cc:1397] T 3dd7d74d54a4439c97eca0d341d736ce P a22832de64344223ba8985121126520f: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20260924 04:14:14.592535 8425 tablet_bootstrap.cc:654] T 3dd7d74d54a4439c97eca0d341d736ce P 13fd4d44db624936bdedc092671ec745: Neither blocks nor log segments found. Creating new log.
I20260924 04:14:14.592569 8426 raft_consensus.cc:348] T 3dd7d74d54a4439c97eca0d341d736ce P a22832de64344223ba8985121126520f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4fdab40943144c38ed46b59ce32516a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } } peers { permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 } } peers { permanent_uuid: "13fd4d44db624936bdedc092671ec745" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36673 } }
I20260924 04:14:14.592679 8426 raft_consensus.cc:374] T 3dd7d74d54a4439c97eca0d341d736ce P a22832de64344223ba8985121126520f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:14:14.592732 8426 raft_consensus.cc:729] T 3dd7d74d54a4439c97eca0d341d736ce P a22832de64344223ba8985121126520f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a22832de64344223ba8985121126520f, State: Initialized, Role: FOLLOWER
I20260924 04:14:14.592845 8426 consensus_queue.cc:260] T 3dd7d74d54a4439c97eca0d341d736ce P a22832de64344223ba8985121126520f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4fdab40943144c38ed46b59ce32516a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } } peers { permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 } } peers { permanent_uuid: "13fd4d44db624936bdedc092671ec745" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36673 } }
I20260924 04:14:14.593741 8426 ts_tablet_manager.cc:1428] T 3dd7d74d54a4439c97eca0d341d736ce P a22832de64344223ba8985121126520f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260924 04:14:14.593760 8425 tablet_bootstrap.cc:492] T 3dd7d74d54a4439c97eca0d341d736ce P 13fd4d44db624936bdedc092671ec745: No bootstrap required, opened a new log
I20260924 04:14:14.593832 8425 ts_tablet_manager.cc:1397] T 3dd7d74d54a4439c97eca0d341d736ce P 13fd4d44db624936bdedc092671ec745: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:14:14.594275 8425 raft_consensus.cc:348] T 3dd7d74d54a4439c97eca0d341d736ce P 13fd4d44db624936bdedc092671ec745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4fdab40943144c38ed46b59ce32516a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } } peers { permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 } } peers { permanent_uuid: "13fd4d44db624936bdedc092671ec745" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36673 } }
I20260924 04:14:14.594414 8425 raft_consensus.cc:374] T 3dd7d74d54a4439c97eca0d341d736ce P 13fd4d44db624936bdedc092671ec745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:14:14.594451 8425 raft_consensus.cc:729] T 3dd7d74d54a4439c97eca0d341d736ce P 13fd4d44db624936bdedc092671ec745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13fd4d44db624936bdedc092671ec745, State: Initialized, Role: FOLLOWER
I20260924 04:14:14.594559 8425 consensus_queue.cc:260] T 3dd7d74d54a4439c97eca0d341d736ce P 13fd4d44db624936bdedc092671ec745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4fdab40943144c38ed46b59ce32516a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } } peers { permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 } } peers { permanent_uuid: "13fd4d44db624936bdedc092671ec745" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36673 } }
I20260924 04:14:14.594817 8425 ts_tablet_manager.cc:1428] T 3dd7d74d54a4439c97eca0d341d736ce P 13fd4d44db624936bdedc092671ec745: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:14:14.595516 7898 catalog_manager.cc:4841] TS 13fd4d44db624936bdedc092671ec745 (127.0.98.3:36673): tablet 29f01da9f7654ae28a6d39f0f9212729 (table test-table [id=caa76cb035464a6da3684fab51d7abb1]) successfully deleted
I20260924 04:14:14.595860 8451 ts_tablet_manager.cc:1905] T 29f01da9f7654ae28a6d39f0f9212729 P c4fdab40943144c38ed46b59ce32516a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:14:14.598623 8451 ts_tablet_manager.cc:1918] T 29f01da9f7654ae28a6d39f0f9212729 P c4fdab40943144c38ed46b59ce32516a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:14:14.598721 8451 log.cc:1196] T 29f01da9f7654ae28a6d39f0f9212729 P c4fdab40943144c38ed46b59ce32516a: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/ts-0/wal/wals/29f01da9f7654ae28a6d39f0f9212729
I20260924 04:14:14.598994 8451 ts_tablet_manager.cc:1939] T 29f01da9f7654ae28a6d39f0f9212729 P c4fdab40943144c38ed46b59ce32516a: Deleting consensus metadata
I20260924 04:14:14.600121 7898 catalog_manager.cc:4841] TS c4fdab40943144c38ed46b59ce32516a (127.0.98.1:42829): tablet 29f01da9f7654ae28a6d39f0f9212729 (table test-table [id=caa76cb035464a6da3684fab51d7abb1]) successfully deleted
I20260924 04:14:14.663079 8431 raft_consensus.cc:482] T 3dd7d74d54a4439c97eca0d341d736ce P a22832de64344223ba8985121126520f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:14:14.663266 8431 raft_consensus.cc:504] T 3dd7d74d54a4439c97eca0d341d736ce P a22832de64344223ba8985121126520f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4fdab40943144c38ed46b59ce32516a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } } peers { permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 } } peers { permanent_uuid: "13fd4d44db624936bdedc092671ec745" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36673 } }
I20260924 04:14:14.664458 8431 leader_election.cc:288] T 3dd7d74d54a4439c97eca0d341d736ce P a22832de64344223ba8985121126520f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4fdab40943144c38ed46b59ce32516a (127.0.98.1:42829), 13fd4d44db624936bdedc092671ec745 (127.0.98.3:36673)
I20260924 04:14:14.668962 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:14.664804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55489 (local address 127.0.98.1:42829)
0924 04:14:14.664941 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:14.664946 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:14.665032 (+ 86us) server_negotiation.cc:407] Connection header received
0924 04:14:14.665245 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:14.665248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:14.665300 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:14.665350 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:14:14.665731 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:14.666314 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:14.667681 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:14.668239 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:14.668368 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:14.668438 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:14.668770 (+ 332us) server_negotiation.cc:299] Negotiation successful
0924 04:14:14.668840 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 04:14:14.669128 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:14.665172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40903 (local address 127.0.98.3:36673)
0924 04:14:14.665278 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:14.665282 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:14.665500 (+ 218us) server_negotiation.cc:407] Connection header received
0924 04:14:14.665585 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:14.665588 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:14.665633 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:14.665718 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:14:14.665975 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:14.666721 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:14.667971 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:14.668466 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:14.668559 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:14.668617 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:14.668929 (+ 312us) server_negotiation.cc:299] Negotiation successful
0924 04:14:14.669007 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 04:14:14.669385 8455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:14.664668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42829 (local address 127.0.98.2:55489)
0924 04:14:14.664942 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:14.664957 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:14.665113 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:14.665493 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:14.665495 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:14.665508 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:14:14.665624 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:14.665630 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:14.666415 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:14.666418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:14.667550 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:14.667561 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:14.668519 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:14.668523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:14.668628 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:14.668652 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 04:14:14.669154 (+ 502us) client_negotiation.cc:240] Negotiation successful
0924 04:14:14.669233 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":95,"threads_started":1}
I20260924 04:14:14.669550 8456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:14.665057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36673 (local address 127.0.98.2:40903)
0924 04:14:14.665409 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:14.665421 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:14.665504 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:14.665736 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:14.665739 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:14.665756 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:14:14.665873 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:14.665879 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:14.666849 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:14.666853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:14.667863 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:14.667870 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:14.668678 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:14.668682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:14.668776 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:14.668811 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 04:14:14.669308 (+ 497us) client_negotiation.cc:240] Negotiation successful
0924 04:14:14.669384 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":98,"threads_started":1}
I20260924 04:14:14.669920 8351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dd7d74d54a4439c97eca0d341d736ce" candidate_uuid: "a22832de64344223ba8985121126520f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13fd4d44db624936bdedc092671ec745" is_pre_election: true
I20260924 04:14:14.669905 8071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dd7d74d54a4439c97eca0d341d736ce" candidate_uuid: "a22832de64344223ba8985121126520f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4fdab40943144c38ed46b59ce32516a" is_pre_election: true
I20260924 04:14:14.670059 8351 raft_consensus.cc:2454] T 3dd7d74d54a4439c97eca0d341d736ce P 13fd4d44db624936bdedc092671ec745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a22832de64344223ba8985121126520f in term 0.
I20260924 04:14:14.670125 8071 raft_consensus.cc:2454] T 3dd7d74d54a4439c97eca0d341d736ce P c4fdab40943144c38ed46b59ce32516a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a22832de64344223ba8985121126520f in term 0.
I20260924 04:14:14.670432 8142 leader_election.cc:302] T 3dd7d74d54a4439c97eca0d341d736ce P a22832de64344223ba8985121126520f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13fd4d44db624936bdedc092671ec745, a22832de64344223ba8985121126520f; no voters:
I20260924 04:14:14.670679 8431 raft_consensus.cc:2781] T 3dd7d74d54a4439c97eca0d341d736ce P a22832de64344223ba8985121126520f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:14:14.670748 8431 raft_consensus.cc:482] T 3dd7d74d54a4439c97eca0d341d736ce P a22832de64344223ba8985121126520f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:14:14.670781 8431 raft_consensus.cc:3037] T 3dd7d74d54a4439c97eca0d341d736ce P a22832de64344223ba8985121126520f [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:14:14.672072 8431 raft_consensus.cc:504] T 3dd7d74d54a4439c97eca0d341d736ce P a22832de64344223ba8985121126520f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4fdab40943144c38ed46b59ce32516a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } } peers { permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 } } peers { permanent_uuid: "13fd4d44db624936bdedc092671ec745" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36673 } }
I20260924 04:14:14.672444 8431 leader_election.cc:288] T 3dd7d74d54a4439c97eca0d341d736ce P a22832de64344223ba8985121126520f [CANDIDATE]: Term 1 election: Requested vote from peers c4fdab40943144c38ed46b59ce32516a (127.0.98.1:42829), 13fd4d44db624936bdedc092671ec745 (127.0.98.3:36673)
I20260924 04:14:14.672858 8071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dd7d74d54a4439c97eca0d341d736ce" candidate_uuid: "a22832de64344223ba8985121126520f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4fdab40943144c38ed46b59ce32516a"
I20260924 04:14:14.672858 8351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dd7d74d54a4439c97eca0d341d736ce" candidate_uuid: "a22832de64344223ba8985121126520f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13fd4d44db624936bdedc092671ec745"
I20260924 04:14:14.672982 8351 raft_consensus.cc:3037] T 3dd7d74d54a4439c97eca0d341d736ce P 13fd4d44db624936bdedc092671ec745 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:14:14.672982 8071 raft_consensus.cc:3037] T 3dd7d74d54a4439c97eca0d341d736ce P c4fdab40943144c38ed46b59ce32516a [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:14:14.674085 8351 raft_consensus.cc:2454] T 3dd7d74d54a4439c97eca0d341d736ce P 13fd4d44db624936bdedc092671ec745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a22832de64344223ba8985121126520f in term 1.
I20260924 04:14:14.674085 8071 raft_consensus.cc:2454] T 3dd7d74d54a4439c97eca0d341d736ce P c4fdab40943144c38ed46b59ce32516a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a22832de64344223ba8985121126520f in term 1.
I20260924 04:14:14.674368 8142 leader_election.cc:302] T 3dd7d74d54a4439c97eca0d341d736ce P a22832de64344223ba8985121126520f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13fd4d44db624936bdedc092671ec745, a22832de64344223ba8985121126520f; no voters:
I20260924 04:14:14.674646 8431 raft_consensus.cc:2781] T 3dd7d74d54a4439c97eca0d341d736ce P a22832de64344223ba8985121126520f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:14:14.674870 8431 raft_consensus.cc:686] T 3dd7d74d54a4439c97eca0d341d736ce P a22832de64344223ba8985121126520f [term 1 LEADER]: Becoming Leader. State: Replica: a22832de64344223ba8985121126520f, State: Running, Role: LEADER
I20260924 04:14:14.675123 8431 consensus_queue.cc:237] T 3dd7d74d54a4439c97eca0d341d736ce P a22832de64344223ba8985121126520f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4fdab40943144c38ed46b59ce32516a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } } peers { permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 } } peers { permanent_uuid: "13fd4d44db624936bdedc092671ec745" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36673 } }
I20260924 04:14:14.677567 7914 catalog_manager.cc:5495] T 3dd7d74d54a4439c97eca0d341d736ce P a22832de64344223ba8985121126520f reported cstate change: term changed from 0 to 1, leader changed from <none> to a22832de64344223ba8985121126520f (127.0.98.2). New cstate: current_term: 1 leader_uuid: "a22832de64344223ba8985121126520f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4fdab40943144c38ed46b59ce32516a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13fd4d44db624936bdedc092671ec745" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36673 } health_report { overall_health: UNKNOWN } } }
Sep 24 04:14:14 dist-test-slave-p401 krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223253, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:14:14.717195 8464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:14.708818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39479 (local address 127.0.0.1:42320)
0924 04:14:14.709078 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:14.709092 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:14.709197 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:14.709435 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:14.709437 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:14.709852 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:14.709944 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:14.709950 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:14.710987 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:14.710990 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:14.711607 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:14.711616 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:14.712161 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:14.712164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:14.712226 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:14.712758 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:14.712770 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:14.714139 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:14:14.715842 (+ 1703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:14.715850 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:14.715854 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:14.716078 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:14.716335 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:14.716337 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:14.716338 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:14.716406 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:14.716755 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:14:14.716759 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:14:14.716871 (+ 112us) client_negotiation.cc:769] Sending connection context
0924 04:14:14.716954 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 04:14:14.717063 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":104,"threads_started":1}
I20260924 04:14:14.717288 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:14.708865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42320 (local address 127.0.98.62:39479)
0924 04:14:14.708996 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:14.709001 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:14.709164 (+ 163us) server_negotiation.cc:407] Connection header received
0924 04:14:14.709267 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:14.709270 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:14.709319 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:14.709421 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:14.710045 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:14.710887 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:14.711697 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:14.712052 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:14.712141 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:14.714264 (+ 2123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:14.714268 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:14.714271 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:14.714294 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:14.715740 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:14.716170 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:14.716174 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:14.716176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:14.716257 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:14.716473 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:14.716476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:14.716478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:14.716673 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:14.716759 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:14.717021 (+ 262us) server_negotiation.cc:299] Negotiation successful
0924 04:14:14.717129 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 04:14:14.720778 7914 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42320:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 04:14:14.721580 7914 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:14:14.726670 8051 tablet_service.cc:1467] Processing CreateTablet for tablet 122b46d661104b9eb65e5532cf9a48e8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=29f72283a0394edca23a9b23ff5c3d47]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 04:14:14.726874 8191 tablet_service.cc:1467] Processing CreateTablet for tablet 122b46d661104b9eb65e5532cf9a48e8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=29f72283a0394edca23a9b23ff5c3d47]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 04:14:14.726874 8331 tablet_service.cc:1467] Processing CreateTablet for tablet 122b46d661104b9eb65e5532cf9a48e8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=29f72283a0394edca23a9b23ff5c3d47]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 04:14:14.727046 8051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 122b46d661104b9eb65e5532cf9a48e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:14:14.727205 8191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 122b46d661104b9eb65e5532cf9a48e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:14:14.727205 8331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 122b46d661104b9eb65e5532cf9a48e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:14:14.729945 8428 tablet_bootstrap.cc:492] T 122b46d661104b9eb65e5532cf9a48e8 P c4fdab40943144c38ed46b59ce32516a: Bootstrap starting.
I20260924 04:14:14.730055 8425 tablet_bootstrap.cc:492] T 122b46d661104b9eb65e5532cf9a48e8 P 13fd4d44db624936bdedc092671ec745: Bootstrap starting.
I20260924 04:14:14.730679 8428 tablet_bootstrap.cc:654] T 122b46d661104b9eb65e5532cf9a48e8 P c4fdab40943144c38ed46b59ce32516a: Neither blocks nor log segments found. Creating new log.
I20260924 04:14:14.730777 8425 tablet_bootstrap.cc:654] T 122b46d661104b9eb65e5532cf9a48e8 P 13fd4d44db624936bdedc092671ec745: Neither blocks nor log segments found. Creating new log.
I20260924 04:14:14.731869 8428 tablet_bootstrap.cc:492] T 122b46d661104b9eb65e5532cf9a48e8 P c4fdab40943144c38ed46b59ce32516a: No bootstrap required, opened a new log
I20260924 04:14:14.731918 8425 tablet_bootstrap.cc:492] T 122b46d661104b9eb65e5532cf9a48e8 P 13fd4d44db624936bdedc092671ec745: No bootstrap required, opened a new log
I20260924 04:14:14.731946 8428 ts_tablet_manager.cc:1397] T 122b46d661104b9eb65e5532cf9a48e8 P c4fdab40943144c38ed46b59ce32516a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:14:14.731977 8425 ts_tablet_manager.cc:1397] T 122b46d661104b9eb65e5532cf9a48e8 P 13fd4d44db624936bdedc092671ec745: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:14:14.732281 8428 raft_consensus.cc:348] T 122b46d661104b9eb65e5532cf9a48e8 P c4fdab40943144c38ed46b59ce32516a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4fdab40943144c38ed46b59ce32516a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } } peers { permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 } } peers { permanent_uuid: "13fd4d44db624936bdedc092671ec745" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36673 } }
I20260924 04:14:14.732309 8425 raft_consensus.cc:348] T 122b46d661104b9eb65e5532cf9a48e8 P 13fd4d44db624936bdedc092671ec745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4fdab40943144c38ed46b59ce32516a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } } peers { permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 } } peers { permanent_uuid: "13fd4d44db624936bdedc092671ec745" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36673 } }
I20260924 04:14:14.732378 8428 raft_consensus.cc:374] T 122b46d661104b9eb65e5532cf9a48e8 P c4fdab40943144c38ed46b59ce32516a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:14:14.732388 8425 raft_consensus.cc:374] T 122b46d661104b9eb65e5532cf9a48e8 P 13fd4d44db624936bdedc092671ec745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:14:14.732409 8425 raft_consensus.cc:729] T 122b46d661104b9eb65e5532cf9a48e8 P 13fd4d44db624936bdedc092671ec745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13fd4d44db624936bdedc092671ec745, State: Initialized, Role: FOLLOWER
I20260924 04:14:14.732409 8428 raft_consensus.cc:729] T 122b46d661104b9eb65e5532cf9a48e8 P c4fdab40943144c38ed46b59ce32516a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4fdab40943144c38ed46b59ce32516a, State: Initialized, Role: FOLLOWER
I20260924 04:14:14.732517 8428 consensus_queue.cc:260] T 122b46d661104b9eb65e5532cf9a48e8 P c4fdab40943144c38ed46b59ce32516a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4fdab40943144c38ed46b59ce32516a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } } peers { permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 } } peers { permanent_uuid: "13fd4d44db624936bdedc092671ec745" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36673 } }
I20260924 04:14:14.732512 8425 consensus_queue.cc:260] T 122b46d661104b9eb65e5532cf9a48e8 P 13fd4d44db624936bdedc092671ec745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4fdab40943144c38ed46b59ce32516a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } } peers { permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 } } peers { permanent_uuid: "13fd4d44db624936bdedc092671ec745" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36673 } }
I20260924 04:14:14.732751 8428 ts_tablet_manager.cc:1428] T 122b46d661104b9eb65e5532cf9a48e8 P c4fdab40943144c38ed46b59ce32516a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:14:14.732751 8425 ts_tablet_manager.cc:1428] T 122b46d661104b9eb65e5532cf9a48e8 P 13fd4d44db624936bdedc092671ec745: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 04:14:14.732745 8432 tablet_replica.cc:439] T 122b46d661104b9eb65e5532cf9a48e8 P 13fd4d44db624936bdedc092671ec745: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4fdab40943144c38ed46b59ce32516a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } } peers { permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 } } peers { permanent_uuid: "13fd4d44db624936bdedc092671ec745" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36673 } } }
I20260924 04:14:14.732925 8432 tablet_replica.cc:442] T 122b46d661104b9eb65e5532cf9a48e8 P 13fd4d44db624936bdedc092671ec745: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:14:14.732957 8446 tablet_replica.cc:439] T 122b46d661104b9eb65e5532cf9a48e8 P c4fdab40943144c38ed46b59ce32516a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4fdab40943144c38ed46b59ce32516a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } } peers { permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 } } peers { permanent_uuid: "13fd4d44db624936bdedc092671ec745" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36673 } } }
I20260924 04:14:14.733065 8446 tablet_replica.cc:442] T 122b46d661104b9eb65e5532cf9a48e8 P c4fdab40943144c38ed46b59ce32516a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:14:14.733740 8426 tablet_bootstrap.cc:492] T 122b46d661104b9eb65e5532cf9a48e8 P a22832de64344223ba8985121126520f: Bootstrap starting.
I20260924 04:14:14.734650 8426 tablet_bootstrap.cc:654] T 122b46d661104b9eb65e5532cf9a48e8 P a22832de64344223ba8985121126520f: Neither blocks nor log segments found. Creating new log.
I20260924 04:14:14.735705 8426 tablet_bootstrap.cc:492] T 122b46d661104b9eb65e5532cf9a48e8 P a22832de64344223ba8985121126520f: No bootstrap required, opened a new log
I20260924 04:14:14.735775 8426 ts_tablet_manager.cc:1397] T 122b46d661104b9eb65e5532cf9a48e8 P a22832de64344223ba8985121126520f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 04:14:14.736126 8426 raft_consensus.cc:348] T 122b46d661104b9eb65e5532cf9a48e8 P a22832de64344223ba8985121126520f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4fdab40943144c38ed46b59ce32516a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } } peers { permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 } } peers { permanent_uuid: "13fd4d44db624936bdedc092671ec745" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36673 } }
I20260924 04:14:14.736214 8426 raft_consensus.cc:374] T 122b46d661104b9eb65e5532cf9a48e8 P a22832de64344223ba8985121126520f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:14:14.736266 8426 raft_consensus.cc:729] T 122b46d661104b9eb65e5532cf9a48e8 P a22832de64344223ba8985121126520f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a22832de64344223ba8985121126520f, State: Initialized, Role: FOLLOWER
I20260924 04:14:14.736393 8426 consensus_queue.cc:260] T 122b46d661104b9eb65e5532cf9a48e8 P a22832de64344223ba8985121126520f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4fdab40943144c38ed46b59ce32516a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } } peers { permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 } } peers { permanent_uuid: "13fd4d44db624936bdedc092671ec745" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36673 } }
I20260924 04:14:14.736606 8426 ts_tablet_manager.cc:1428] T 122b46d661104b9eb65e5532cf9a48e8 P a22832de64344223ba8985121126520f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 04:14:14.736759 8431 tablet_replica.cc:439] T 122b46d661104b9eb65e5532cf9a48e8 P a22832de64344223ba8985121126520f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4fdab40943144c38ed46b59ce32516a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } } peers { permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 } } peers { permanent_uuid: "13fd4d44db624936bdedc092671ec745" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36673 } } }
I20260924 04:14:14.736850 8431 tablet_replica.cc:442] T 122b46d661104b9eb65e5532cf9a48e8 P a22832de64344223ba8985121126520f: This TxnStatusTablet replica's current role is: FOLLOWER
W20260924 04:14:14.835278 8118 tablet.cc:2369] T 122b46d661104b9eb65e5532cf9a48e8 P c4fdab40943144c38ed46b59ce32516a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:14:14.999893 8446 raft_consensus.cc:482] T 122b46d661104b9eb65e5532cf9a48e8 P c4fdab40943144c38ed46b59ce32516a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:14:15.000031 8446 raft_consensus.cc:504] T 122b46d661104b9eb65e5532cf9a48e8 P c4fdab40943144c38ed46b59ce32516a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4fdab40943144c38ed46b59ce32516a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } } peers { permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 } } peers { permanent_uuid: "13fd4d44db624936bdedc092671ec745" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36673 } }
I20260924 04:14:15.000455 8446 leader_election.cc:288] T 122b46d661104b9eb65e5532cf9a48e8 P c4fdab40943144c38ed46b59ce32516a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a22832de64344223ba8985121126520f (127.0.98.2:44637), 13fd4d44db624936bdedc092671ec745 (127.0.98.3:36673)
I20260924 04:14:15.000901 8211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "122b46d661104b9eb65e5532cf9a48e8" candidate_uuid: "c4fdab40943144c38ed46b59ce32516a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a22832de64344223ba8985121126520f" is_pre_election: true
I20260924 04:14:15.000902 8351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "122b46d661104b9eb65e5532cf9a48e8" candidate_uuid: "c4fdab40943144c38ed46b59ce32516a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13fd4d44db624936bdedc092671ec745" is_pre_election: true
I20260924 04:14:15.001044 8211 raft_consensus.cc:2454] T 122b46d661104b9eb65e5532cf9a48e8 P a22832de64344223ba8985121126520f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4fdab40943144c38ed46b59ce32516a in term 0.
I20260924 04:14:15.001052 8351 raft_consensus.cc:2454] T 122b46d661104b9eb65e5532cf9a48e8 P 13fd4d44db624936bdedc092671ec745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4fdab40943144c38ed46b59ce32516a in term 0.
I20260924 04:14:15.001340 8004 leader_election.cc:302] T 122b46d661104b9eb65e5532cf9a48e8 P c4fdab40943144c38ed46b59ce32516a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a22832de64344223ba8985121126520f, c4fdab40943144c38ed46b59ce32516a; no voters:
I20260924 04:14:15.001507 8446 raft_consensus.cc:2781] T 122b46d661104b9eb65e5532cf9a48e8 P c4fdab40943144c38ed46b59ce32516a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:14:15.001560 8446 raft_consensus.cc:482] T 122b46d661104b9eb65e5532cf9a48e8 P c4fdab40943144c38ed46b59ce32516a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:14:15.001596 8446 raft_consensus.cc:3037] T 122b46d661104b9eb65e5532cf9a48e8 P c4fdab40943144c38ed46b59ce32516a [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:14:15.002514 8446 raft_consensus.cc:504] T 122b46d661104b9eb65e5532cf9a48e8 P c4fdab40943144c38ed46b59ce32516a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4fdab40943144c38ed46b59ce32516a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } } peers { permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 } } peers { permanent_uuid: "13fd4d44db624936bdedc092671ec745" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36673 } }
I20260924 04:14:15.002789 8446 leader_election.cc:288] T 122b46d661104b9eb65e5532cf9a48e8 P c4fdab40943144c38ed46b59ce32516a [CANDIDATE]: Term 1 election: Requested vote from peers a22832de64344223ba8985121126520f (127.0.98.2:44637), 13fd4d44db624936bdedc092671ec745 (127.0.98.3:36673)
I20260924 04:14:15.003139 8351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "122b46d661104b9eb65e5532cf9a48e8" candidate_uuid: "c4fdab40943144c38ed46b59ce32516a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13fd4d44db624936bdedc092671ec745"
I20260924 04:14:15.003136 8211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "122b46d661104b9eb65e5532cf9a48e8" candidate_uuid: "c4fdab40943144c38ed46b59ce32516a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a22832de64344223ba8985121126520f"
I20260924 04:14:15.003271 8351 raft_consensus.cc:3037] T 122b46d661104b9eb65e5532cf9a48e8 P 13fd4d44db624936bdedc092671ec745 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:14:15.003332 8211 raft_consensus.cc:3037] T 122b46d661104b9eb65e5532cf9a48e8 P a22832de64344223ba8985121126520f [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:14:15.004107 8351 raft_consensus.cc:2454] T 122b46d661104b9eb65e5532cf9a48e8 P 13fd4d44db624936bdedc092671ec745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4fdab40943144c38ed46b59ce32516a in term 1.
I20260924 04:14:15.004247 8211 raft_consensus.cc:2454] T 122b46d661104b9eb65e5532cf9a48e8 P a22832de64344223ba8985121126520f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4fdab40943144c38ed46b59ce32516a in term 1.
I20260924 04:14:15.004387 8002 leader_election.cc:302] T 122b46d661104b9eb65e5532cf9a48e8 P c4fdab40943144c38ed46b59ce32516a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13fd4d44db624936bdedc092671ec745, c4fdab40943144c38ed46b59ce32516a; no voters:
I20260924 04:14:15.004565 8446 raft_consensus.cc:2781] T 122b46d661104b9eb65e5532cf9a48e8 P c4fdab40943144c38ed46b59ce32516a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:14:15.004649 8446 raft_consensus.cc:686] T 122b46d661104b9eb65e5532cf9a48e8 P c4fdab40943144c38ed46b59ce32516a [term 1 LEADER]: Becoming Leader. State: Replica: c4fdab40943144c38ed46b59ce32516a, State: Running, Role: LEADER
I20260924 04:14:15.004786 8446 consensus_queue.cc:237] T 122b46d661104b9eb65e5532cf9a48e8 P c4fdab40943144c38ed46b59ce32516a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4fdab40943144c38ed46b59ce32516a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } } peers { permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 } } peers { permanent_uuid: "13fd4d44db624936bdedc092671ec745" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36673 } }
I20260924 04:14:15.005378 8433 tablet_replica.cc:439] T 122b46d661104b9eb65e5532cf9a48e8 P c4fdab40943144c38ed46b59ce32516a: TxnStatusTablet state changed. Reason: New leader c4fdab40943144c38ed46b59ce32516a. Latest consensus state: current_term: 1 leader_uuid: "c4fdab40943144c38ed46b59ce32516a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4fdab40943144c38ed46b59ce32516a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } } peers { permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 } } peers { permanent_uuid: "13fd4d44db624936bdedc092671ec745" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36673 } } }
I20260924 04:14:15.005486 8433 tablet_replica.cc:442] T 122b46d661104b9eb65e5532cf9a48e8 P c4fdab40943144c38ed46b59ce32516a: This TxnStatusTablet replica's current role is: LEADER
I20260924 04:14:15.005750 8469 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 04:14:15.005903 8469 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 04:14:15.006386 7910 catalog_manager.cc:5495] T 122b46d661104b9eb65e5532cf9a48e8 P c4fdab40943144c38ed46b59ce32516a reported cstate change: term changed from 0 to 1, leader changed from <none> to c4fdab40943144c38ed46b59ce32516a (127.0.98.1). New cstate: current_term: 1 leader_uuid: "c4fdab40943144c38ed46b59ce32516a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4fdab40943144c38ed46b59ce32516a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13fd4d44db624936bdedc092671ec745" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36673 } health_report { overall_health: UNKNOWN } } }
I20260924 04:14:15.030267 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:15.027308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53054 (local address 127.0.98.1:42829)
0924 04:14:15.027434 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:15.027438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:15.027452 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:14:15.027607 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:15.027610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:15.027660 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:15.027719 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:14:15.027960 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:15.028460 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:15.029275 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:15.029494 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:15.029578 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:15.029837 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:14:15.029909 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:14:15.029966 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:15.030071 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 04:14:15.030126 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 04:14:15.030267 8464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:15.027208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42829 (local address 127.0.0.1:53054)
0924 04:14:15.027370 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:15.027387 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:15.027494 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:15.027744 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:15.027746 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:15.027756 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:14:15.027866 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:15.027871 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:15.028556 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:15.028559 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:15.029168 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:15.029176 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:15.029586 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:15.029589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:15.029673 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:15.029715 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:14:15.029980 (+ 265us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:14:15.029994 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 04:14:15.030052 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 04:14:15.030141 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 04:14:15.032814 8351 raft_consensus.cc:1261] T 122b46d661104b9eb65e5532cf9a48e8 P 13fd4d44db624936bdedc092671ec745 [term 1 FOLLOWER]: Refusing update from remote peer c4fdab40943144c38ed46b59ce32516a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:14:15.032902 8211 raft_consensus.cc:1261] T 122b46d661104b9eb65e5532cf9a48e8 P a22832de64344223ba8985121126520f [term 1 FOLLOWER]: Refusing update from remote peer c4fdab40943144c38ed46b59ce32516a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:14:15.033274 8446 consensus_queue.cc:1035] T 122b46d661104b9eb65e5532cf9a48e8 P c4fdab40943144c38ed46b59ce32516a [LEADER]: Connected to new peer: Peer: permanent_uuid: "13fd4d44db624936bdedc092671ec745" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:14:15.033452 8433 consensus_queue.cc:1035] T 122b46d661104b9eb65e5532cf9a48e8 P c4fdab40943144c38ed46b59ce32516a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:14:15.034739 8431 tablet_replica.cc:439] T 122b46d661104b9eb65e5532cf9a48e8 P a22832de64344223ba8985121126520f: TxnStatusTablet state changed. Reason: New leader c4fdab40943144c38ed46b59ce32516a. Latest consensus state: current_term: 1 leader_uuid: "c4fdab40943144c38ed46b59ce32516a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4fdab40943144c38ed46b59ce32516a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } } peers { permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 } } peers { permanent_uuid: "13fd4d44db624936bdedc092671ec745" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36673 } } }
I20260924 04:14:15.034747 8432 tablet_replica.cc:439] T 122b46d661104b9eb65e5532cf9a48e8 P 13fd4d44db624936bdedc092671ec745: TxnStatusTablet state changed. Reason: New leader c4fdab40943144c38ed46b59ce32516a. Latest consensus state: current_term: 1 leader_uuid: "c4fdab40943144c38ed46b59ce32516a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4fdab40943144c38ed46b59ce32516a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } } peers { permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 } } peers { permanent_uuid: "13fd4d44db624936bdedc092671ec745" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36673 } } }
I20260924 04:14:15.034849 8431 tablet_replica.cc:442] T 122b46d661104b9eb65e5532cf9a48e8 P a22832de64344223ba8985121126520f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:14:15.034854 8432 tablet_replica.cc:442] T 122b46d661104b9eb65e5532cf9a48e8 P 13fd4d44db624936bdedc092671ec745: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:14:15.037252 8433 tablet_replica.cc:439] T 122b46d661104b9eb65e5532cf9a48e8 P c4fdab40943144c38ed46b59ce32516a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c4fdab40943144c38ed46b59ce32516a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4fdab40943144c38ed46b59ce32516a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } } peers { permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 } } peers { permanent_uuid: "13fd4d44db624936bdedc092671ec745" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36673 } } }
I20260924 04:14:15.037367 8433 tablet_replica.cc:442] T 122b46d661104b9eb65e5532cf9a48e8 P c4fdab40943144c38ed46b59ce32516a: This TxnStatusTablet replica's current role is: LEADER
I20260924 04:14:15.037782 8441 mvcc.cc:205] Tried to move back new op lower bound from 7332754452610711552 to 7332754452500783104. Current Snapshot: MvccSnapshot[applied={T|T < 7332754452610711552}]
I20260924 04:14:15.037943 8432 tablet_replica.cc:439] T 122b46d661104b9eb65e5532cf9a48e8 P 13fd4d44db624936bdedc092671ec745: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c4fdab40943144c38ed46b59ce32516a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4fdab40943144c38ed46b59ce32516a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } } peers { permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 } } peers { permanent_uuid: "13fd4d44db624936bdedc092671ec745" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36673 } } }
I20260924 04:14:15.038029 8432 tablet_replica.cc:442] T 122b46d661104b9eb65e5532cf9a48e8 P 13fd4d44db624936bdedc092671ec745: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:14:15.039755 8439 mvcc.cc:205] Tried to move back new op lower bound from 7332754452610711552 to 7332754452500783104. Current Snapshot: MvccSnapshot[applied={T|T < 7332754452610711552 or (T in {7332754452610711552})}]
I20260924 04:14:15.040334 8433 tablet_replica.cc:439] T 122b46d661104b9eb65e5532cf9a48e8 P c4fdab40943144c38ed46b59ce32516a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c4fdab40943144c38ed46b59ce32516a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4fdab40943144c38ed46b59ce32516a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } } peers { permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 } } peers { permanent_uuid: "13fd4d44db624936bdedc092671ec745" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36673 } } }
I20260924 04:14:15.040427 8433 tablet_replica.cc:442] T 122b46d661104b9eb65e5532cf9a48e8 P c4fdab40943144c38ed46b59ce32516a: This TxnStatusTablet replica's current role is: LEADER
I20260924 04:14:15.042184 8431 tablet_replica.cc:439] T 122b46d661104b9eb65e5532cf9a48e8 P a22832de64344223ba8985121126520f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c4fdab40943144c38ed46b59ce32516a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4fdab40943144c38ed46b59ce32516a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } } peers { permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 } } peers { permanent_uuid: "13fd4d44db624936bdedc092671ec745" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36673 } } }
I20260924 04:14:15.042294 8431 tablet_replica.cc:442] T 122b46d661104b9eb65e5532cf9a48e8 P a22832de64344223ba8985121126520f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:14:15.046737 8474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:15.041999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56886 (local address 127.0.98.2:44637)
0924 04:14:15.042263 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:15.042271 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:15.042284 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:14:15.042367 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:15.042369 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:15.042439 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:15.042826 (+ 387us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:14:15.042918 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:15.043724 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:15.044920 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:15.045284 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:15.045391 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:15.045647 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:14:15.046183 (+ 536us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:14:15.046291 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:15.046393 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 04:14:15.046451 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"spinlock_wait_cycles":3840,"thread_start_us":111,"threads_started":1}
I20260924 04:14:15.057885 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:15.054087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33249 (local address 127.0.98.1:42829)
0924 04:14:15.054206 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:15.054211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:15.054390 (+ 179us) server_negotiation.cc:407] Connection header received
0924 04:14:15.054496 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:15.054499 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:15.054549 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:15.054619 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:14:15.054822 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:15.055614 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:15.056786 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:15.057116 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:15.057205 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:15.057485 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:14:15.057557 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:14:15.057610 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:15.057705 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 04:14:15.057756 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 04:14:15.057914 8477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:15.054006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42829 (local address 127.0.98.2:33249)
0924 04:14:15.054286 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:15.054317 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:15.054417 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:15.054632 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:15.054635 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:15.054642 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:14:15.054739 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:15.054745 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:15.055718 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:15.055722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:15.056674 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:15.056684 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:15.057220 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:15.057224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:15.057315 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:15.057364 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:14:15.057628 (+ 264us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:14:15.057634 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 04:14:15.057688 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 04:14:15.057763 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":93,"threads_started":1}
I20260924 04:14:15.064579 8351 raft_consensus.cc:1261] T 3dd7d74d54a4439c97eca0d341d736ce P 13fd4d44db624936bdedc092671ec745 [term 1 FOLLOWER]: Refusing update from remote peer a22832de64344223ba8985121126520f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:14:15.064991 8431 consensus_queue.cc:1035] T 3dd7d74d54a4439c97eca0d341d736ce P a22832de64344223ba8985121126520f [LEADER]: Connected to new peer: Peer: permanent_uuid: "13fd4d44db624936bdedc092671ec745" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:14:15.065076 8071 raft_consensus.cc:1261] T 3dd7d74d54a4439c97eca0d341d736ce P c4fdab40943144c38ed46b59ce32516a [term 1 FOLLOWER]: Refusing update from remote peer a22832de64344223ba8985121126520f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:14:15.065490 8431 consensus_queue.cc:1035] T 3dd7d74d54a4439c97eca0d341d736ce P a22832de64344223ba8985121126520f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4fdab40943144c38ed46b59ce32516a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:14:15.086283 8474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:15.081774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42037 (local address 127.0.98.2:44637)
0924 04:14:15.081876 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:15.081881 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:15.082424 (+ 543us) server_negotiation.cc:407] Connection header received
0924 04:14:15.082599 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:15.082602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:15.082651 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:15.082692 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:14:15.083078 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:15.083626 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:15.084867 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:15.085203 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:15.085268 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:15.085640 (+ 372us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:14:15.085706 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:14:15.085759 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:15.086140 (+ 381us) server_negotiation.cc:299] Negotiation successful
0924 04:14:15.086182 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 04:14:15.086369 8481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:15.081743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44637 (local address 127.0.98.1:42037)
0924 04:14:15.082346 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:15.082360 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:15.082493 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:15.082812 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:15.082815 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:15.082822 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:14:15.082970 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:15.082976 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:15.083714 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:15.083717 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:15.084735 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:15.084746 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:15.085391 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:15.085395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:15.085479 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:15.085525 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:14:15.086056 (+ 531us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:14:15.086062 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 04:14:15.086133 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 04:14:15.086214 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":96,"threads_started":1}
I20260924 04:14:15.112350 7914 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:14:15.112514 7914 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:14:15.114578 7914 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 32a4954b64f244c6aa3aee7bdcf49528: Sending DeleteTablet for 3 replicas of tablet 3dd7d74d54a4439c97eca0d341d736ce
I20260924 04:14:15.115198 8191 tablet_service.cc:1514] Processing DeleteTablet for tablet 3dd7d74d54a4439c97eca0d341d736ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:14:15 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56870
I20260924 04:14:15.115269 8331 tablet_service.cc:1514] Processing DeleteTablet for tablet 3dd7d74d54a4439c97eca0d341d736ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:14:15 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38136
I20260924 04:14:15.115816 8051 tablet_service.cc:1514] Processing DeleteTablet for tablet 3dd7d74d54a4439c97eca0d341d736ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:14:15 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53036
I20260924 04:14:15.116190 8487 tablet_replica.cc:330] T 3dd7d74d54a4439c97eca0d341d736ce P c4fdab40943144c38ed46b59ce32516a: stopping tablet replica
I20260924 04:14:15.116344 8487 raft_consensus.cc:2229] T 3dd7d74d54a4439c97eca0d341d736ce P c4fdab40943144c38ed46b59ce32516a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:14:15.116506 8487 raft_consensus.cc:2258] T 3dd7d74d54a4439c97eca0d341d736ce P c4fdab40943144c38ed46b59ce32516a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:14:15.117278 8487 ts_tablet_manager.cc:1905] T 3dd7d74d54a4439c97eca0d341d736ce P c4fdab40943144c38ed46b59ce32516a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:14:15.120450 8487 ts_tablet_manager.cc:1918] T 3dd7d74d54a4439c97eca0d341d736ce P c4fdab40943144c38ed46b59ce32516a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:14:15.120605 8487 log.cc:1196] T 3dd7d74d54a4439c97eca0d341d736ce P c4fdab40943144c38ed46b59ce32516a: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/ts-0/wal/wals/3dd7d74d54a4439c97eca0d341d736ce
I20260924 04:14:15.120919 8487 ts_tablet_manager.cc:1939] T 3dd7d74d54a4439c97eca0d341d736ce P c4fdab40943144c38ed46b59ce32516a: Deleting consensus metadata
I20260924 04:14:15.121732 7898 catalog_manager.cc:4841] TS c4fdab40943144c38ed46b59ce32516a (127.0.98.1:42829): tablet 3dd7d74d54a4439c97eca0d341d736ce (table test-table [id=a822988f3c504eff8e8f5f68294be8e9]) successfully deleted
I20260924 04:14:15.123430 8486 tablet_replica.cc:330] T 3dd7d74d54a4439c97eca0d341d736ce P 13fd4d44db624936bdedc092671ec745: stopping tablet replica
I20260924 04:14:15.123575 8486 raft_consensus.cc:2229] T 3dd7d74d54a4439c97eca0d341d736ce P 13fd4d44db624936bdedc092671ec745 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:14:15.123708 8486 raft_consensus.cc:2258] T 3dd7d74d54a4439c97eca0d341d736ce P 13fd4d44db624936bdedc092671ec745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:14:15.127820 8486 ts_tablet_manager.cc:1905] T 3dd7d74d54a4439c97eca0d341d736ce P 13fd4d44db624936bdedc092671ec745: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 24 04:14:15 dist-test-slave-p401 krb5kdc[7869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260924 04:14:15.130774 8486 ts_tablet_manager.cc:1918] T 3dd7d74d54a4439c97eca0d341d736ce P 13fd4d44db624936bdedc092671ec745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:14:15.130867 8486 log.cc:1196] T 3dd7d74d54a4439c97eca0d341d736ce P 13fd4d44db624936bdedc092671ec745: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/ts-2/wal/wals/3dd7d74d54a4439c97eca0d341d736ce
I20260924 04:14:15.131093 8486 ts_tablet_manager.cc:1939] T 3dd7d74d54a4439c97eca0d341d736ce P 13fd4d44db624936bdedc092671ec745: Deleting consensus metadata
I20260924 04:14:15.131877 7898 catalog_manager.cc:4841] TS 13fd4d44db624936bdedc092671ec745 (127.0.98.3:36673): tablet 3dd7d74d54a4439c97eca0d341d736ce (table test-table [id=a822988f3c504eff8e8f5f68294be8e9]) successfully deleted
I20260924 04:14:15.132213 8485 tablet_replica.cc:330] T 3dd7d74d54a4439c97eca0d341d736ce P a22832de64344223ba8985121126520f: stopping tablet replica
I20260924 04:14:15.132346 8485 raft_consensus.cc:2229] T 3dd7d74d54a4439c97eca0d341d736ce P a22832de64344223ba8985121126520f [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:14:15.132661 8485 raft_consensus.cc:2258] T 3dd7d74d54a4439c97eca0d341d736ce P a22832de64344223ba8985121126520f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:14:15.133322 8485 ts_tablet_manager.cc:1905] T 3dd7d74d54a4439c97eca0d341d736ce P a22832de64344223ba8985121126520f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:14:15.136018 8485 ts_tablet_manager.cc:1918] T 3dd7d74d54a4439c97eca0d341d736ce P a22832de64344223ba8985121126520f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:14:15.136088 8485 log.cc:1196] T 3dd7d74d54a4439c97eca0d341d736ce P a22832de64344223ba8985121126520f: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/ts-1/wal/wals/3dd7d74d54a4439c97eca0d341d736ce
I20260924 04:14:15.136322 8485 ts_tablet_manager.cc:1939] T 3dd7d74d54a4439c97eca0d341d736ce P a22832de64344223ba8985121126520f: Deleting consensus metadata
I20260924 04:14:15.137053 7900 catalog_manager.cc:4841] TS a22832de64344223ba8985121126520f (127.0.98.2:44637): tablet 3dd7d74d54a4439c97eca0d341d736ce (table test-table [id=a822988f3c504eff8e8f5f68294be8e9]) successfully deleted
Sep 24 04:14:15 dist-test-slave-p401 krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:14:15 dist-test-slave-p401 krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 04:14:15.143862 7963 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:15.137825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42334 (local address 127.0.98.62:39479)
0924 04:14:15.137946 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:15.137950 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:15.138185 (+ 235us) server_negotiation.cc:407] Connection header received
0924 04:14:15.138302 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:15.138305 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:15.138355 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:15.138422 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:15.139105 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:15.139728 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:15.140582 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:15.140807 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:15.140898 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:15.143743 (+ 2845us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42334: BlockingRecv error: recv got EOF from 127.0.0.1:42334 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 24 04:14:15 dist-test-slave-p401 krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:14:15.153323 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:15.145446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42348 (local address 127.0.98.62:39479)
0924 04:14:15.145556 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:15.145560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:15.145752 (+ 192us) server_negotiation.cc:407] Connection header received
0924 04:14:15.145854 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:15.145858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:15.145901 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:15.145962 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:15.146568 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:15.147183 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:15.148012 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:15.148237 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:15.148319 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:15.150496 (+ 2177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:15.150499 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:15.150501 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:15.150525 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:15.151853 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:15.152226 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:15.152230 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:15.152231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:15.152319 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:15.152559 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:15.152562 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:15.152564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:15.152743 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:15.152856 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:15.153064 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 04:14:15.153165 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 04:14:15.155937 7914 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42348:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:14:15.156363 7914 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:14:15.160971 8051 tablet_service.cc:1467] Processing CreateTablet for tablet e45cbe00ff36437e80e0cdf25f49fd8c (DEFAULT_TABLE table=test-table [id=9cb927c561cc4846ba64c7d329c85ff6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:14:15.161185 8191 tablet_service.cc:1467] Processing CreateTablet for tablet e45cbe00ff36437e80e0cdf25f49fd8c (DEFAULT_TABLE table=test-table [id=9cb927c561cc4846ba64c7d329c85ff6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:14:15.161279 8051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e45cbe00ff36437e80e0cdf25f49fd8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:14:15.161420 8191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e45cbe00ff36437e80e0cdf25f49fd8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:14:15.163758 8428 tablet_bootstrap.cc:492] T e45cbe00ff36437e80e0cdf25f49fd8c P c4fdab40943144c38ed46b59ce32516a: Bootstrap starting.
I20260924 04:14:15.164034 8331 tablet_service.cc:1467] Processing CreateTablet for tablet e45cbe00ff36437e80e0cdf25f49fd8c (DEFAULT_TABLE table=test-table [id=9cb927c561cc4846ba64c7d329c85ff6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:14:15.164294 8331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e45cbe00ff36437e80e0cdf25f49fd8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:14:15.164561 8428 tablet_bootstrap.cc:654] T e45cbe00ff36437e80e0cdf25f49fd8c P c4fdab40943144c38ed46b59ce32516a: Neither blocks nor log segments found. Creating new log.
I20260924 04:14:15.166220 8425 tablet_bootstrap.cc:492] T e45cbe00ff36437e80e0cdf25f49fd8c P 13fd4d44db624936bdedc092671ec745: Bootstrap starting.
I20260924 04:14:15.167071 8425 tablet_bootstrap.cc:654] T e45cbe00ff36437e80e0cdf25f49fd8c P 13fd4d44db624936bdedc092671ec745: Neither blocks nor log segments found. Creating new log.
I20260924 04:14:15.167992 8426 tablet_bootstrap.cc:492] T e45cbe00ff36437e80e0cdf25f49fd8c P a22832de64344223ba8985121126520f: Bootstrap starting.
I20260924 04:14:15.168877 8428 tablet_bootstrap.cc:492] T e45cbe00ff36437e80e0cdf25f49fd8c P c4fdab40943144c38ed46b59ce32516a: No bootstrap required, opened a new log
I20260924 04:14:15.168962 8426 tablet_bootstrap.cc:654] T e45cbe00ff36437e80e0cdf25f49fd8c P a22832de64344223ba8985121126520f: Neither blocks nor log segments found. Creating new log.
I20260924 04:14:15.168967 8428 ts_tablet_manager.cc:1397] T e45cbe00ff36437e80e0cdf25f49fd8c P c4fdab40943144c38ed46b59ce32516a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260924 04:14:15.169409 8428 raft_consensus.cc:348] T e45cbe00ff36437e80e0cdf25f49fd8c P c4fdab40943144c38ed46b59ce32516a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4fdab40943144c38ed46b59ce32516a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } } peers { permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 } } peers { permanent_uuid: "13fd4d44db624936bdedc092671ec745" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36673 } }
I20260924 04:14:15.169517 8428 raft_consensus.cc:374] T e45cbe00ff36437e80e0cdf25f49fd8c P c4fdab40943144c38ed46b59ce32516a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:14:15.169567 8428 raft_consensus.cc:729] T e45cbe00ff36437e80e0cdf25f49fd8c P c4fdab40943144c38ed46b59ce32516a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4fdab40943144c38ed46b59ce32516a, State: Initialized, Role: FOLLOWER
I20260924 04:14:15.169699 8428 consensus_queue.cc:260] T e45cbe00ff36437e80e0cdf25f49fd8c P c4fdab40943144c38ed46b59ce32516a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4fdab40943144c38ed46b59ce32516a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } } peers { permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 } } peers { permanent_uuid: "13fd4d44db624936bdedc092671ec745" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36673 } }
I20260924 04:14:15.169961 8428 ts_tablet_manager.cc:1428] T e45cbe00ff36437e80e0cdf25f49fd8c P c4fdab40943144c38ed46b59ce32516a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:14:15.170121 8426 tablet_bootstrap.cc:492] T e45cbe00ff36437e80e0cdf25f49fd8c P a22832de64344223ba8985121126520f: No bootstrap required, opened a new log
I20260924 04:14:15.170197 8426 ts_tablet_manager.cc:1397] T e45cbe00ff36437e80e0cdf25f49fd8c P a22832de64344223ba8985121126520f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260924 04:14:15.170593 8426 raft_consensus.cc:348] T e45cbe00ff36437e80e0cdf25f49fd8c P a22832de64344223ba8985121126520f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4fdab40943144c38ed46b59ce32516a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } } peers { permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 } } peers { permanent_uuid: "13fd4d44db624936bdedc092671ec745" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36673 } }
I20260924 04:14:15.170708 8426 raft_consensus.cc:374] T e45cbe00ff36437e80e0cdf25f49fd8c P a22832de64344223ba8985121126520f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:14:15.170739 8426 raft_consensus.cc:729] T e45cbe00ff36437e80e0cdf25f49fd8c P a22832de64344223ba8985121126520f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a22832de64344223ba8985121126520f, State: Initialized, Role: FOLLOWER
I20260924 04:14:15.170846 8426 consensus_queue.cc:260] T e45cbe00ff36437e80e0cdf25f49fd8c P a22832de64344223ba8985121126520f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4fdab40943144c38ed46b59ce32516a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } } peers { permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 } } peers { permanent_uuid: "13fd4d44db624936bdedc092671ec745" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36673 } }
I20260924 04:14:15.171082 8426 ts_tablet_manager.cc:1428] T e45cbe00ff36437e80e0cdf25f49fd8c P a22832de64344223ba8985121126520f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 04:14:15.176118 8425 tablet_bootstrap.cc:492] T e45cbe00ff36437e80e0cdf25f49fd8c P 13fd4d44db624936bdedc092671ec745: No bootstrap required, opened a new log
I20260924 04:14:15.176236 8425 ts_tablet_manager.cc:1397] T e45cbe00ff36437e80e0cdf25f49fd8c P 13fd4d44db624936bdedc092671ec745: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.001s
I20260924 04:14:15.176710 8425 raft_consensus.cc:348] T e45cbe00ff36437e80e0cdf25f49fd8c P 13fd4d44db624936bdedc092671ec745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4fdab40943144c38ed46b59ce32516a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } } peers { permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 } } peers { permanent_uuid: "13fd4d44db624936bdedc092671ec745" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36673 } }
I20260924 04:14:15.176818 8425 raft_consensus.cc:374] T e45cbe00ff36437e80e0cdf25f49fd8c P 13fd4d44db624936bdedc092671ec745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:14:15.176847 8425 raft_consensus.cc:729] T e45cbe00ff36437e80e0cdf25f49fd8c P 13fd4d44db624936bdedc092671ec745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13fd4d44db624936bdedc092671ec745, State: Initialized, Role: FOLLOWER
I20260924 04:14:15.176994 8425 consensus_queue.cc:260] T e45cbe00ff36437e80e0cdf25f49fd8c P 13fd4d44db624936bdedc092671ec745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4fdab40943144c38ed46b59ce32516a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } } peers { permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 } } peers { permanent_uuid: "13fd4d44db624936bdedc092671ec745" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36673 } }
I20260924 04:14:15.177256 8425 ts_tablet_manager.cc:1428] T e45cbe00ff36437e80e0cdf25f49fd8c P 13fd4d44db624936bdedc092671ec745: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 04:14:15.271472 8432 raft_consensus.cc:482] T e45cbe00ff36437e80e0cdf25f49fd8c P 13fd4d44db624936bdedc092671ec745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:14:15.271621 8432 raft_consensus.cc:504] T e45cbe00ff36437e80e0cdf25f49fd8c P 13fd4d44db624936bdedc092671ec745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4fdab40943144c38ed46b59ce32516a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } } peers { permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 } } peers { permanent_uuid: "13fd4d44db624936bdedc092671ec745" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36673 } }
I20260924 04:14:15.272644 8432 leader_election.cc:288] T e45cbe00ff36437e80e0cdf25f49fd8c P 13fd4d44db624936bdedc092671ec745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4fdab40943144c38ed46b59ce32516a (127.0.98.1:42829), a22832de64344223ba8985121126520f (127.0.98.2:44637)
I20260924 04:14:15.276767 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:15.272965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54603 (local address 127.0.98.1:42829)
0924 04:14:15.273081 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:15.273086 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:15.273215 (+ 129us) server_negotiation.cc:407] Connection header received
0924 04:14:15.273410 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:15.273412 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:15.273454 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:15.273490 (+ 36us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:14:15.273843 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:15.274593 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:15.275681 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:15.276197 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:15.276295 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:15.276381 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:15.276565 (+ 184us) server_negotiation.cc:299] Negotiation successful
0924 04:14:15.276643 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 04:14:15.277129 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:15.272852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42829 (local address 127.0.98.3:54603)
0924 04:14:15.273140 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:15.273153 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:15.273289 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:15.273604 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:15.273607 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:15.273637 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:14:15.273705 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:15.273709 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:15.274717 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:15.274719 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:15.275564 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:15.275572 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:15.276334 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:15.276337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:15.276426 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:15.276452 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 04:14:15.276913 (+ 461us) client_negotiation.cc:240] Negotiation successful
0924 04:14:15.276990 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"spinlock_wait_cycles":384,"thread_start_us":82,"threads_started":1}
I20260924 04:14:15.277630 8071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e45cbe00ff36437e80e0cdf25f49fd8c" candidate_uuid: "13fd4d44db624936bdedc092671ec745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4fdab40943144c38ed46b59ce32516a" is_pre_election: true
I20260924 04:14:15.277791 8071 raft_consensus.cc:2454] T e45cbe00ff36437e80e0cdf25f49fd8c P c4fdab40943144c38ed46b59ce32516a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13fd4d44db624936bdedc092671ec745 in term 0.
I20260924 04:14:15.277859 8474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:15.273208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40427 (local address 127.0.98.2:44637)
0924 04:14:15.273324 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:15.273331 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:15.273369 (+ 38us) server_negotiation.cc:407] Connection header received
0924 04:14:15.273421 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:15.273424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:15.273473 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:15.273548 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:14:15.274744 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:15.275417 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:15.276417 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:15.276951 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:15.277024 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:15.277082 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:15.277666 (+ 584us) server_negotiation.cc:299] Negotiation successful
0924 04:14:15.277741 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 04:14:15.278169 8503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:15.272895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44637 (local address 127.0.98.3:40427)
0924 04:14:15.273211 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:15.273223 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:15.273297 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:15.273574 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:15.273577 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:15.273597 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:14:15.273701 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:15.273707 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:15.275524 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:15.275528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:15.276271 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:15.276281 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:15.277399 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:15.277402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:15.277484 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:15.277552 (+ 68us) client_negotiation.cc:769] Sending connection context
0924 04:14:15.277968 (+ 416us) client_negotiation.cc:240] Negotiation successful
0924 04:14:15.278038 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"spinlock_wait_cycles":4608,"thread_start_us":58,"threads_started":1}
I20260924 04:14:15.278208 8282 leader_election.cc:302] T e45cbe00ff36437e80e0cdf25f49fd8c P 13fd4d44db624936bdedc092671ec745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13fd4d44db624936bdedc092671ec745, c4fdab40943144c38ed46b59ce32516a; no voters:
I20260924 04:14:15.278442 8432 raft_consensus.cc:2781] T e45cbe00ff36437e80e0cdf25f49fd8c P 13fd4d44db624936bdedc092671ec745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:14:15.278491 8432 raft_consensus.cc:482] T e45cbe00ff36437e80e0cdf25f49fd8c P 13fd4d44db624936bdedc092671ec745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:14:15.278515 8432 raft_consensus.cc:3037] T e45cbe00ff36437e80e0cdf25f49fd8c P 13fd4d44db624936bdedc092671ec745 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:14:15.278563 8211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e45cbe00ff36437e80e0cdf25f49fd8c" candidate_uuid: "13fd4d44db624936bdedc092671ec745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a22832de64344223ba8985121126520f" is_pre_election: true
I20260924 04:14:15.278700 8211 raft_consensus.cc:2454] T e45cbe00ff36437e80e0cdf25f49fd8c P a22832de64344223ba8985121126520f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13fd4d44db624936bdedc092671ec745 in term 0.
I20260924 04:14:15.279511 8432 raft_consensus.cc:504] T e45cbe00ff36437e80e0cdf25f49fd8c P 13fd4d44db624936bdedc092671ec745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4fdab40943144c38ed46b59ce32516a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } } peers { permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 } } peers { permanent_uuid: "13fd4d44db624936bdedc092671ec745" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36673 } }
I20260924 04:14:15.279820 8432 leader_election.cc:288] T e45cbe00ff36437e80e0cdf25f49fd8c P 13fd4d44db624936bdedc092671ec745 [CANDIDATE]: Term 1 election: Requested vote from peers c4fdab40943144c38ed46b59ce32516a (127.0.98.1:42829), a22832de64344223ba8985121126520f (127.0.98.2:44637)
I20260924 04:14:15.280097 8071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e45cbe00ff36437e80e0cdf25f49fd8c" candidate_uuid: "13fd4d44db624936bdedc092671ec745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4fdab40943144c38ed46b59ce32516a"
I20260924 04:14:15.280123 8211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e45cbe00ff36437e80e0cdf25f49fd8c" candidate_uuid: "13fd4d44db624936bdedc092671ec745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a22832de64344223ba8985121126520f"
I20260924 04:14:15.280212 8071 raft_consensus.cc:3037] T e45cbe00ff36437e80e0cdf25f49fd8c P c4fdab40943144c38ed46b59ce32516a [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:14:15.280212 8211 raft_consensus.cc:3037] T e45cbe00ff36437e80e0cdf25f49fd8c P a22832de64344223ba8985121126520f [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:14:15.281227 8071 raft_consensus.cc:2454] T e45cbe00ff36437e80e0cdf25f49fd8c P c4fdab40943144c38ed46b59ce32516a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13fd4d44db624936bdedc092671ec745 in term 1.
I20260924 04:14:15.281227 8211 raft_consensus.cc:2454] T e45cbe00ff36437e80e0cdf25f49fd8c P a22832de64344223ba8985121126520f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13fd4d44db624936bdedc092671ec745 in term 1.
I20260924 04:14:15.281473 8284 leader_election.cc:302] T e45cbe00ff36437e80e0cdf25f49fd8c P 13fd4d44db624936bdedc092671ec745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13fd4d44db624936bdedc092671ec745, a22832de64344223ba8985121126520f; no voters:
I20260924 04:14:15.281617 8432 raft_consensus.cc:2781] T e45cbe00ff36437e80e0cdf25f49fd8c P 13fd4d44db624936bdedc092671ec745 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:14:15.281782 8432 raft_consensus.cc:686] T e45cbe00ff36437e80e0cdf25f49fd8c P 13fd4d44db624936bdedc092671ec745 [term 1 LEADER]: Becoming Leader. State: Replica: 13fd4d44db624936bdedc092671ec745, State: Running, Role: LEADER
I20260924 04:14:15.282016 8432 consensus_queue.cc:237] T e45cbe00ff36437e80e0cdf25f49fd8c P 13fd4d44db624936bdedc092671ec745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4fdab40943144c38ed46b59ce32516a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } } peers { permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 } } peers { permanent_uuid: "13fd4d44db624936bdedc092671ec745" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36673 } }
I20260924 04:14:15.283964 7914 catalog_manager.cc:5495] T e45cbe00ff36437e80e0cdf25f49fd8c P 13fd4d44db624936bdedc092671ec745 reported cstate change: term changed from 0 to 1, leader changed from <none> to 13fd4d44db624936bdedc092671ec745 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "13fd4d44db624936bdedc092671ec745" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4fdab40943144c38ed46b59ce32516a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13fd4d44db624936bdedc092671ec745" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36673 } health_report { overall_health: HEALTHY } } }
I20260924 04:14:15.297361 8508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:15.293843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38140 (local address 127.0.98.3:36673)
0924 04:14:15.294115 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:15.294123 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:15.294134 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:14:15.294201 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:15.294203 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:15.294252 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:15.294373 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:14:15.294597 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:15.295125 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:15.295972 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:15.296207 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:15.296290 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:15.296546 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:14:15.296993 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:14:15.297092 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:15.297159 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 04:14:15.297210 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":103,"threads_started":1}
I20260924 04:14:15.299728 8071 raft_consensus.cc:1261] T e45cbe00ff36437e80e0cdf25f49fd8c P c4fdab40943144c38ed46b59ce32516a [term 1 FOLLOWER]: Refusing update from remote peer 13fd4d44db624936bdedc092671ec745: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:14:15.299746 8211 raft_consensus.cc:1261] T e45cbe00ff36437e80e0cdf25f49fd8c P a22832de64344223ba8985121126520f [term 1 FOLLOWER]: Refusing update from remote peer 13fd4d44db624936bdedc092671ec745: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:14:15.300184 8432 consensus_queue.cc:1035] T e45cbe00ff36437e80e0cdf25f49fd8c P 13fd4d44db624936bdedc092671ec745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:14:15.300355 8505 consensus_queue.cc:1035] T e45cbe00ff36437e80e0cdf25f49fd8c P 13fd4d44db624936bdedc092671ec745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4fdab40943144c38ed46b59ce32516a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:14:15.311724 7910 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:14:15.311864 7910 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:14:15.313808 7910 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 32a4954b64f244c6aa3aee7bdcf49528: Sending DeleteTablet for 3 replicas of tablet e45cbe00ff36437e80e0cdf25f49fd8c
I20260924 04:14:15.314275 8051 tablet_service.cc:1514] Processing DeleteTablet for tablet e45cbe00ff36437e80e0cdf25f49fd8c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:14:15 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53036
I20260924 04:14:15.314365 8191 tablet_service.cc:1514] Processing DeleteTablet for tablet e45cbe00ff36437e80e0cdf25f49fd8c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:14:15 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56870
I20260924 04:14:15.314459 8487 tablet_replica.cc:330] T e45cbe00ff36437e80e0cdf25f49fd8c P c4fdab40943144c38ed46b59ce32516a: stopping tablet replica
I20260924 04:14:15.314544 8485 tablet_replica.cc:330] T e45cbe00ff36437e80e0cdf25f49fd8c P a22832de64344223ba8985121126520f: stopping tablet replica
I20260924 04:14:15.314577 8487 raft_consensus.cc:2229] T e45cbe00ff36437e80e0cdf25f49fd8c P c4fdab40943144c38ed46b59ce32516a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:14:15.314580 8331 tablet_service.cc:1514] Processing DeleteTablet for tablet e45cbe00ff36437e80e0cdf25f49fd8c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:14:15 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38136
I20260924 04:14:15.314651 8485 raft_consensus.cc:2229] T e45cbe00ff36437e80e0cdf25f49fd8c P a22832de64344223ba8985121126520f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:14:15.314707 8487 raft_consensus.cc:2258] T e45cbe00ff36437e80e0cdf25f49fd8c P c4fdab40943144c38ed46b59ce32516a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:14:15.314771 8485 raft_consensus.cc:2258] T e45cbe00ff36437e80e0cdf25f49fd8c P a22832de64344223ba8985121126520f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:14:15.314857 8486 tablet_replica.cc:330] T e45cbe00ff36437e80e0cdf25f49fd8c P 13fd4d44db624936bdedc092671ec745: stopping tablet replica
I20260924 04:14:15.314945 8486 raft_consensus.cc:2229] T e45cbe00ff36437e80e0cdf25f49fd8c P 13fd4d44db624936bdedc092671ec745 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:14:15.315220 8486 raft_consensus.cc:2258] T e45cbe00ff36437e80e0cdf25f49fd8c P 13fd4d44db624936bdedc092671ec745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:14:15.315768 8487 ts_tablet_manager.cc:1905] T e45cbe00ff36437e80e0cdf25f49fd8c P c4fdab40943144c38ed46b59ce32516a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:14:15.315764 8485 ts_tablet_manager.cc:1905] T e45cbe00ff36437e80e0cdf25f49fd8c P a22832de64344223ba8985121126520f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:14:15.315793 8486 ts_tablet_manager.cc:1905] T e45cbe00ff36437e80e0cdf25f49fd8c P 13fd4d44db624936bdedc092671ec745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:14:15.318068 8486 ts_tablet_manager.cc:1918] T e45cbe00ff36437e80e0cdf25f49fd8c P 13fd4d44db624936bdedc092671ec745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:14:15.318068 8487 ts_tablet_manager.cc:1918] T e45cbe00ff36437e80e0cdf25f49fd8c P c4fdab40943144c38ed46b59ce32516a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:14:15.318133 8487 log.cc:1196] T e45cbe00ff36437e80e0cdf25f49fd8c P c4fdab40943144c38ed46b59ce32516a: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/ts-0/wal/wals/e45cbe00ff36437e80e0cdf25f49fd8c
I20260924 04:14:15.318133 8486 log.cc:1196] T e45cbe00ff36437e80e0cdf25f49fd8c P 13fd4d44db624936bdedc092671ec745: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/ts-2/wal/wals/e45cbe00ff36437e80e0cdf25f49fd8c
I20260924 04:14:15.318168 8485 ts_tablet_manager.cc:1918] T e45cbe00ff36437e80e0cdf25f49fd8c P a22832de64344223ba8985121126520f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:14:15.318236 8485 log.cc:1196] T e45cbe00ff36437e80e0cdf25f49fd8c P a22832de64344223ba8985121126520f: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/ts-1/wal/wals/e45cbe00ff36437e80e0cdf25f49fd8c
I20260924 04:14:15.318372 8487 ts_tablet_manager.cc:1939] T e45cbe00ff36437e80e0cdf25f49fd8c P c4fdab40943144c38ed46b59ce32516a: Deleting consensus metadata
I20260924 04:14:15.318372 8486 ts_tablet_manager.cc:1939] T e45cbe00ff36437e80e0cdf25f49fd8c P 13fd4d44db624936bdedc092671ec745: Deleting consensus metadata
I20260924 04:14:15.318473 8485 ts_tablet_manager.cc:1939] T e45cbe00ff36437e80e0cdf25f49fd8c P a22832de64344223ba8985121126520f: Deleting consensus metadata
I20260924 04:14:15.318935 7910 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42348:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 04:14:15.319100 7900 catalog_manager.cc:4841] TS a22832de64344223ba8985121126520f (127.0.98.2:44637): tablet e45cbe00ff36437e80e0cdf25f49fd8c (table test-table [id=9cb927c561cc4846ba64c7d329c85ff6]) successfully deleted
W20260924 04:14:15.319339 7910 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:14:15.320011 7898 catalog_manager.cc:4841] TS 13fd4d44db624936bdedc092671ec745 (127.0.98.3:36673): tablet e45cbe00ff36437e80e0cdf25f49fd8c (table test-table [id=9cb927c561cc4846ba64c7d329c85ff6]) successfully deleted
I20260924 04:14:15.320601 7898 catalog_manager.cc:4841] TS c4fdab40943144c38ed46b59ce32516a (127.0.98.1:42829): tablet e45cbe00ff36437e80e0cdf25f49fd8c (table test-table [id=9cb927c561cc4846ba64c7d329c85ff6]) successfully deleted
I20260924 04:14:15.323627 8051 tablet_service.cc:1467] Processing CreateTablet for tablet bac3a9e6e93149689ee523a31daf27f8 (DEFAULT_TABLE table=test-table [id=65646707016d48929a750c4a1a22cce1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:14:15.323848 8191 tablet_service.cc:1467] Processing CreateTablet for tablet bac3a9e6e93149689ee523a31daf27f8 (DEFAULT_TABLE table=test-table [id=65646707016d48929a750c4a1a22cce1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:14:15.323915 8051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bac3a9e6e93149689ee523a31daf27f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:14:15.324064 8191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bac3a9e6e93149689ee523a31daf27f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:14:15.326121 8428 tablet_bootstrap.cc:492] T bac3a9e6e93149689ee523a31daf27f8 P c4fdab40943144c38ed46b59ce32516a: Bootstrap starting.
I20260924 04:14:15.326906 8428 tablet_bootstrap.cc:654] T bac3a9e6e93149689ee523a31daf27f8 P c4fdab40943144c38ed46b59ce32516a: Neither blocks nor log segments found. Creating new log.
I20260924 04:14:15.327386 8426 tablet_bootstrap.cc:492] T bac3a9e6e93149689ee523a31daf27f8 P a22832de64344223ba8985121126520f: Bootstrap starting.
I20260924 04:14:15.328032 8426 tablet_bootstrap.cc:654] T bac3a9e6e93149689ee523a31daf27f8 P a22832de64344223ba8985121126520f: Neither blocks nor log segments found. Creating new log.
I20260924 04:14:15.328588 8331 tablet_service.cc:1467] Processing CreateTablet for tablet bac3a9e6e93149689ee523a31daf27f8 (DEFAULT_TABLE table=test-table [id=65646707016d48929a750c4a1a22cce1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:14:15.328807 8331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bac3a9e6e93149689ee523a31daf27f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:14:15.329006 8426 tablet_bootstrap.cc:492] T bac3a9e6e93149689ee523a31daf27f8 P a22832de64344223ba8985121126520f: No bootstrap required, opened a new log
I20260924 04:14:15.329079 8426 ts_tablet_manager.cc:1397] T bac3a9e6e93149689ee523a31daf27f8 P a22832de64344223ba8985121126520f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:14:15.329463 8426 raft_consensus.cc:348] T bac3a9e6e93149689ee523a31daf27f8 P a22832de64344223ba8985121126520f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4fdab40943144c38ed46b59ce32516a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } } peers { permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 } } peers { permanent_uuid: "13fd4d44db624936bdedc092671ec745" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36673 } }
I20260924 04:14:15.329568 8426 raft_consensus.cc:374] T bac3a9e6e93149689ee523a31daf27f8 P a22832de64344223ba8985121126520f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:14:15.329597 8426 raft_consensus.cc:729] T bac3a9e6e93149689ee523a31daf27f8 P a22832de64344223ba8985121126520f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a22832de64344223ba8985121126520f, State: Initialized, Role: FOLLOWER
I20260924 04:14:15.329716 8426 consensus_queue.cc:260] T bac3a9e6e93149689ee523a31daf27f8 P a22832de64344223ba8985121126520f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4fdab40943144c38ed46b59ce32516a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } } peers { permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 } } peers { permanent_uuid: "13fd4d44db624936bdedc092671ec745" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36673 } }
I20260924 04:14:15.329978 8426 ts_tablet_manager.cc:1428] T bac3a9e6e93149689ee523a31daf27f8 P a22832de64344223ba8985121126520f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 04:14:15.330737 8425 tablet_bootstrap.cc:492] T bac3a9e6e93149689ee523a31daf27f8 P 13fd4d44db624936bdedc092671ec745: Bootstrap starting.
I20260924 04:14:15.331557 8425 tablet_bootstrap.cc:654] T bac3a9e6e93149689ee523a31daf27f8 P 13fd4d44db624936bdedc092671ec745: Neither blocks nor log segments found. Creating new log.
I20260924 04:14:15.332718 8428 tablet_bootstrap.cc:492] T bac3a9e6e93149689ee523a31daf27f8 P c4fdab40943144c38ed46b59ce32516a: No bootstrap required, opened a new log
I20260924 04:14:15.332819 8428 ts_tablet_manager.cc:1397] T bac3a9e6e93149689ee523a31daf27f8 P c4fdab40943144c38ed46b59ce32516a: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20260924 04:14:15.333199 8428 raft_consensus.cc:348] T bac3a9e6e93149689ee523a31daf27f8 P c4fdab40943144c38ed46b59ce32516a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4fdab40943144c38ed46b59ce32516a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } } peers { permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 } } peers { permanent_uuid: "13fd4d44db624936bdedc092671ec745" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36673 } }
I20260924 04:14:15.333298 8428 raft_consensus.cc:374] T bac3a9e6e93149689ee523a31daf27f8 P c4fdab40943144c38ed46b59ce32516a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:14:15.333319 8428 raft_consensus.cc:729] T bac3a9e6e93149689ee523a31daf27f8 P c4fdab40943144c38ed46b59ce32516a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4fdab40943144c38ed46b59ce32516a, State: Initialized, Role: FOLLOWER
I20260924 04:14:15.333421 8428 consensus_queue.cc:260] T bac3a9e6e93149689ee523a31daf27f8 P c4fdab40943144c38ed46b59ce32516a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4fdab40943144c38ed46b59ce32516a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } } peers { permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 } } peers { permanent_uuid: "13fd4d44db624936bdedc092671ec745" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36673 } }
I20260924 04:14:15.333727 8428 ts_tablet_manager.cc:1428] T bac3a9e6e93149689ee523a31daf27f8 P c4fdab40943144c38ed46b59ce32516a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 04:14:15.334115 8425 tablet_bootstrap.cc:492] T bac3a9e6e93149689ee523a31daf27f8 P 13fd4d44db624936bdedc092671ec745: No bootstrap required, opened a new log
I20260924 04:14:15.334177 8425 ts_tablet_manager.cc:1397] T bac3a9e6e93149689ee523a31daf27f8 P 13fd4d44db624936bdedc092671ec745: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 04:14:15.334571 8425 raft_consensus.cc:348] T bac3a9e6e93149689ee523a31daf27f8 P 13fd4d44db624936bdedc092671ec745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4fdab40943144c38ed46b59ce32516a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } } peers { permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 } } peers { permanent_uuid: "13fd4d44db624936bdedc092671ec745" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36673 } }
I20260924 04:14:15.334698 8425 raft_consensus.cc:374] T bac3a9e6e93149689ee523a31daf27f8 P 13fd4d44db624936bdedc092671ec745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:14:15.334738 8425 raft_consensus.cc:729] T bac3a9e6e93149689ee523a31daf27f8 P 13fd4d44db624936bdedc092671ec745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13fd4d44db624936bdedc092671ec745, State: Initialized, Role: FOLLOWER
I20260924 04:14:15.334848 8425 consensus_queue.cc:260] T bac3a9e6e93149689ee523a31daf27f8 P 13fd4d44db624936bdedc092671ec745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4fdab40943144c38ed46b59ce32516a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } } peers { permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 } } peers { permanent_uuid: "13fd4d44db624936bdedc092671ec745" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36673 } }
I20260924 04:14:15.335098 8425 ts_tablet_manager.cc:1428] T bac3a9e6e93149689ee523a31daf27f8 P 13fd4d44db624936bdedc092671ec745: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 04:14:15.514197 8511 raft_consensus.cc:482] T bac3a9e6e93149689ee523a31daf27f8 P 13fd4d44db624936bdedc092671ec745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:14:15.514345 8511 raft_consensus.cc:504] T bac3a9e6e93149689ee523a31daf27f8 P 13fd4d44db624936bdedc092671ec745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4fdab40943144c38ed46b59ce32516a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } } peers { permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 } } peers { permanent_uuid: "13fd4d44db624936bdedc092671ec745" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36673 } }
I20260924 04:14:15.514773 8511 leader_election.cc:288] T bac3a9e6e93149689ee523a31daf27f8 P 13fd4d44db624936bdedc092671ec745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4fdab40943144c38ed46b59ce32516a (127.0.98.1:42829), a22832de64344223ba8985121126520f (127.0.98.2:44637)
I20260924 04:14:15.515278 8211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bac3a9e6e93149689ee523a31daf27f8" candidate_uuid: "13fd4d44db624936bdedc092671ec745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a22832de64344223ba8985121126520f" is_pre_election: true
I20260924 04:14:15.515278 8071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bac3a9e6e93149689ee523a31daf27f8" candidate_uuid: "13fd4d44db624936bdedc092671ec745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4fdab40943144c38ed46b59ce32516a" is_pre_election: true
I20260924 04:14:15.515429 8211 raft_consensus.cc:2454] T bac3a9e6e93149689ee523a31daf27f8 P a22832de64344223ba8985121126520f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13fd4d44db624936bdedc092671ec745 in term 0.
I20260924 04:14:15.515429 8071 raft_consensus.cc:2454] T bac3a9e6e93149689ee523a31daf27f8 P c4fdab40943144c38ed46b59ce32516a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13fd4d44db624936bdedc092671ec745 in term 0.
I20260924 04:14:15.515702 8284 leader_election.cc:302] T bac3a9e6e93149689ee523a31daf27f8 P 13fd4d44db624936bdedc092671ec745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 13fd4d44db624936bdedc092671ec745, a22832de64344223ba8985121126520f, c4fdab40943144c38ed46b59ce32516a; no voters:
I20260924 04:14:15.515942 8511 raft_consensus.cc:2781] T bac3a9e6e93149689ee523a31daf27f8 P 13fd4d44db624936bdedc092671ec745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:14:15.516017 8511 raft_consensus.cc:482] T bac3a9e6e93149689ee523a31daf27f8 P 13fd4d44db624936bdedc092671ec745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:14:15.516065 8511 raft_consensus.cc:3037] T bac3a9e6e93149689ee523a31daf27f8 P 13fd4d44db624936bdedc092671ec745 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:14:15.517162 8511 raft_consensus.cc:504] T bac3a9e6e93149689ee523a31daf27f8 P 13fd4d44db624936bdedc092671ec745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4fdab40943144c38ed46b59ce32516a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } } peers { permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 } } peers { permanent_uuid: "13fd4d44db624936bdedc092671ec745" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36673 } }
I20260924 04:14:15.517503 8511 leader_election.cc:288] T bac3a9e6e93149689ee523a31daf27f8 P 13fd4d44db624936bdedc092671ec745 [CANDIDATE]: Term 1 election: Requested vote from peers c4fdab40943144c38ed46b59ce32516a (127.0.98.1:42829), a22832de64344223ba8985121126520f (127.0.98.2:44637)
I20260924 04:14:15.517827 8071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bac3a9e6e93149689ee523a31daf27f8" candidate_uuid: "13fd4d44db624936bdedc092671ec745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4fdab40943144c38ed46b59ce32516a"
I20260924 04:14:15.517827 8211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bac3a9e6e93149689ee523a31daf27f8" candidate_uuid: "13fd4d44db624936bdedc092671ec745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a22832de64344223ba8985121126520f"
I20260924 04:14:15.517967 8211 raft_consensus.cc:3037] T bac3a9e6e93149689ee523a31daf27f8 P a22832de64344223ba8985121126520f [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:14:15.517964 8071 raft_consensus.cc:3037] T bac3a9e6e93149689ee523a31daf27f8 P c4fdab40943144c38ed46b59ce32516a [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:14:15.519110 8211 raft_consensus.cc:2454] T bac3a9e6e93149689ee523a31daf27f8 P a22832de64344223ba8985121126520f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13fd4d44db624936bdedc092671ec745 in term 1.
I20260924 04:14:15.519119 8071 raft_consensus.cc:2454] T bac3a9e6e93149689ee523a31daf27f8 P c4fdab40943144c38ed46b59ce32516a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13fd4d44db624936bdedc092671ec745 in term 1.
I20260924 04:14:15.519434 8284 leader_election.cc:302] T bac3a9e6e93149689ee523a31daf27f8 P 13fd4d44db624936bdedc092671ec745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 13fd4d44db624936bdedc092671ec745, a22832de64344223ba8985121126520f, c4fdab40943144c38ed46b59ce32516a; no voters:
I20260924 04:14:15.519609 8511 raft_consensus.cc:2781] T bac3a9e6e93149689ee523a31daf27f8 P 13fd4d44db624936bdedc092671ec745 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:14:15.519695 8511 raft_consensus.cc:686] T bac3a9e6e93149689ee523a31daf27f8 P 13fd4d44db624936bdedc092671ec745 [term 1 LEADER]: Becoming Leader. State: Replica: 13fd4d44db624936bdedc092671ec745, State: Running, Role: LEADER
I20260924 04:14:15.519840 8511 consensus_queue.cc:237] T bac3a9e6e93149689ee523a31daf27f8 P 13fd4d44db624936bdedc092671ec745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4fdab40943144c38ed46b59ce32516a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } } peers { permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 } } peers { permanent_uuid: "13fd4d44db624936bdedc092671ec745" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36673 } }
I20260924 04:14:15.521423 7914 catalog_manager.cc:5495] T bac3a9e6e93149689ee523a31daf27f8 P 13fd4d44db624936bdedc092671ec745 reported cstate change: term changed from 0 to 1, leader changed from <none> to 13fd4d44db624936bdedc092671ec745 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "13fd4d44db624936bdedc092671ec745" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4fdab40943144c38ed46b59ce32516a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13fd4d44db624936bdedc092671ec745" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36673 } health_report { overall_health: HEALTHY } } }
I20260924 04:14:15.586169 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:15.581338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59651 (local address 127.0.98.1:42829)
0924 04:14:15.581463 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:15.581468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:15.583514 (+ 2046us) server_negotiation.cc:407] Connection header received
0924 04:14:15.583633 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:15.583635 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:15.583678 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:15.583737 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:14:15.583948 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:15.584434 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:15.585238 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:15.585442 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:15.585525 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:15.585766 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:14:15.585842 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:14:15.585897 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:15.585982 (+ 85us) server_negotiation.cc:299] Negotiation successful
0924 04:14:15.586029 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 04:14:15.586179 8519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:15.581237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42829 (local address 127.0.98.3:59651)
0924 04:14:15.583430 (+ 2193us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:15.583443 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:15.583546 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:15.583751 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:15.583754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:15.583761 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:14:15.583861 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:15.583865 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:15.584529 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:15.584532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:15.585139 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:15.585146 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:15.585532 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:15.585536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:15.585611 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:15.585652 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:14:15.585912 (+ 260us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:14:15.585918 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 04:14:15.585974 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 04:14:15.586056 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2100,"thread_start_us":90,"threads_started":1}
I20260924 04:14:15.591404 8071 raft_consensus.cc:1261] T bac3a9e6e93149689ee523a31daf27f8 P c4fdab40943144c38ed46b59ce32516a [term 1 FOLLOWER]: Refusing update from remote peer 13fd4d44db624936bdedc092671ec745: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:14:15.591414 8211 raft_consensus.cc:1261] T bac3a9e6e93149689ee523a31daf27f8 P a22832de64344223ba8985121126520f [term 1 FOLLOWER]: Refusing update from remote peer 13fd4d44db624936bdedc092671ec745: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:14:15.591781 8511 consensus_queue.cc:1035] T bac3a9e6e93149689ee523a31daf27f8 P 13fd4d44db624936bdedc092671ec745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4fdab40943144c38ed46b59ce32516a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:14:15.591946 8432 consensus_queue.cc:1035] T bac3a9e6e93149689ee523a31daf27f8 P 13fd4d44db624936bdedc092671ec745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a22832de64344223ba8985121126520f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:14:15.607501 8524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:15.604254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36673 (local address 127.0.98.1:51643)
0924 04:14:15.604659 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:15.604671 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:15.604761 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:15.604956 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:15.604959 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:15.604966 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:14:15.605064 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:15.605070 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:15.605748 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:15.605752 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:15.606355 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:15.606363 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:15.606833 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:15.606836 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:15.606923 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:15.606951 (+ 28us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:14:15.607185 (+ 234us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:14:15.607191 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 04:14:15.607287 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 04:14:15.607378 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":73,"threads_started":1}
I20260924 04:14:15.607488 8508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:15.604313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51643 (local address 127.0.98.3:36673)
0924 04:14:15.604427 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:15.604430 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:15.604741 (+ 311us) server_negotiation.cc:407] Connection header received
0924 04:14:15.604840 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:15.604843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:15.604887 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:15.604942 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:14:15.605153 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:15.605653 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:15.606460 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:15.606729 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:15.606824 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:15.607049 (+ 225us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:14:15.607118 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:14:15.607174 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:15.607299 (+ 125us) server_negotiation.cc:299] Negotiation successful
0924 04:14:15.607352 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:14:15.625831 7914 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:14:15.625973 7914 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:14:15.627686 7914 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 32a4954b64f244c6aa3aee7bdcf49528: Sending DeleteTablet for 3 replicas of tablet bac3a9e6e93149689ee523a31daf27f8
I20260924 04:14:15.628089 8051 tablet_service.cc:1514] Processing DeleteTablet for tablet bac3a9e6e93149689ee523a31daf27f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:14:15 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53036
I20260924 04:14:15.628170 8331 tablet_service.cc:1514] Processing DeleteTablet for tablet bac3a9e6e93149689ee523a31daf27f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:14:15 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38136
I20260924 04:14:15.628278 8191 tablet_service.cc:1514] Processing DeleteTablet for tablet bac3a9e6e93149689ee523a31daf27f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:14:15 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56870
I20260924 04:14:15.628345 8486 tablet_replica.cc:330] T bac3a9e6e93149689ee523a31daf27f8 P 13fd4d44db624936bdedc092671ec745: stopping tablet replica
I20260924 04:14:15.628444 8487 tablet_replica.cc:330] T bac3a9e6e93149689ee523a31daf27f8 P c4fdab40943144c38ed46b59ce32516a: stopping tablet replica
I20260924 04:14:15.628454 8486 raft_consensus.cc:2229] T bac3a9e6e93149689ee523a31daf27f8 P 13fd4d44db624936bdedc092671ec745 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:14:15.628566 8487 raft_consensus.cc:2229] T bac3a9e6e93149689ee523a31daf27f8 P c4fdab40943144c38ed46b59ce32516a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:14:15.628618 8486 raft_consensus.cc:2258] T bac3a9e6e93149689ee523a31daf27f8 P 13fd4d44db624936bdedc092671ec745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:14:15.628664 8487 raft_consensus.cc:2258] T bac3a9e6e93149689ee523a31daf27f8 P c4fdab40943144c38ed46b59ce32516a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:14:15.628739 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 7986
I20260924 04:14:15.628813 8485 tablet_replica.cc:330] T bac3a9e6e93149689ee523a31daf27f8 P a22832de64344223ba8985121126520f: stopping tablet replica
I20260924 04:14:15.628914 8485 raft_consensus.cc:2229] T bac3a9e6e93149689ee523a31daf27f8 P a22832de64344223ba8985121126520f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:14:15.628991 8485 raft_consensus.cc:2258] T bac3a9e6e93149689ee523a31daf27f8 P a22832de64344223ba8985121126520f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:14:15.630326 8485 ts_tablet_manager.cc:1905] T bac3a9e6e93149689ee523a31daf27f8 P a22832de64344223ba8985121126520f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:14:15.631695 8486 ts_tablet_manager.cc:1905] T bac3a9e6e93149689ee523a31daf27f8 P 13fd4d44db624936bdedc092671ec745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:14:15.632556 8485 ts_tablet_manager.cc:1918] T bac3a9e6e93149689ee523a31daf27f8 P a22832de64344223ba8985121126520f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:14:15.632633 8485 log.cc:1196] T bac3a9e6e93149689ee523a31daf27f8 P a22832de64344223ba8985121126520f: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/ts-1/wal/wals/bac3a9e6e93149689ee523a31daf27f8
I20260924 04:14:15.632866 8485 ts_tablet_manager.cc:1939] T bac3a9e6e93149689ee523a31daf27f8 P a22832de64344223ba8985121126520f: Deleting consensus metadata
I20260924 04:14:15.633508 7900 catalog_manager.cc:4841] TS a22832de64344223ba8985121126520f (127.0.98.2:44637): tablet bac3a9e6e93149689ee523a31daf27f8 (table test-table [id=65646707016d48929a750c4a1a22cce1]) successfully deleted
I20260924 04:14:15.633850 8486 ts_tablet_manager.cc:1918] T bac3a9e6e93149689ee523a31daf27f8 P 13fd4d44db624936bdedc092671ec745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:14:15.633908 8486 log.cc:1196] T bac3a9e6e93149689ee523a31daf27f8 P 13fd4d44db624936bdedc092671ec745: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790223216265354-392-0/minicluster-data/ts-2/wal/wals/bac3a9e6e93149689ee523a31daf27f8
I20260924 04:14:15.634094 8486 ts_tablet_manager.cc:1939] T bac3a9e6e93149689ee523a31daf27f8 P 13fd4d44db624936bdedc092671ec745: Deleting consensus metadata
I20260924 04:14:15.634646 7898 catalog_manager.cc:4841] TS 13fd4d44db624936bdedc092671ec745 (127.0.98.3:36673): tablet bac3a9e6e93149689ee523a31daf27f8 (table test-table [id=65646707016d48929a750c4a1a22cce1]) successfully deleted
W20260924 04:14:15.636852 7898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42829 (error 108)
I20260924 04:14:15.637022 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 8126
I20260924 04:14:15.638927 8527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:15.638590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42829 (local address 127.0.0.1:53056)
0924 04:14:15.638789 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:15.638850 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":60,"threads_started":1}
W20260924 04:14:15.639045 7898 catalog_manager.cc:4584] TS c4fdab40943144c38ed46b59ce32516a (127.0.98.1:42829): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bac3a9e6e93149689ee523a31daf27f8: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42829: connect: Connection refused (error 111)
I20260924 04:14:15.642460 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 8266
I20260924 04:14:15.647768 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 7882
2026-09-24T04:14:15Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3314 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790223216265354-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:14:15 dist-test-slave-p401 krb5kdc[8530](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:14:15 dist-test-slave-p401 krb5kdc[8530](info): set up 2 sockets
Sep 24 04:14:15 dist-test-slave-p401 krb5kdc[8530](info): commencing operation
krb5kdc: starting...
W20260924 04:14:16.599689 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790223216265354-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790223216265354-392-0/krb5kdc/test-user.keytab.
Sep 24 04:14:16 dist-test-slave-p401 krb5kdc[8530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:14:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:14:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:14:16.713694 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790223216265354-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790223216265354-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790223216265354-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39629
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39629
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:14:16.820730 8543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:14:16.820996 8543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:14:16.821051 8543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:14:16.824980 8543 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:14:16.825037 8543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:14:16.825062 8543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:14:16.825083 8543 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:14:16.825105 8543 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:14:16.830200 8543 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790223216265354-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790223216265354-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39629
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790223216265354-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:14:16.831192 8543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:14:16.832206 8543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:14:16.840061 8543 server_base.cc:1034] running on GCE node
W20260924 04:14:16.840098 8548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:16.840009 8549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:16.840009 8551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:16.840725 8543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:14:16.841723 8543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:14:16.842871 8543 hybrid_clock.cc:648] HybridClock initialized: now 1790223256842858 us; error 26 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-24T04:14:16Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1203 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:14:16 dist-test-slave-p401 krb5kdc[8556](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:14:16 dist-test-slave-p401 krb5kdc[8556](info): set up 2 sockets
Sep 24 04:14:16 dist-test-slave-p401 krb5kdc[8556](info): commencing operation
krb5kdc: starting...
W20260924 04:14:17.799561 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/krb5kdc/test-user.keytab.
Sep 24 04:14:17 dist-test-slave-p401 krb5kdc[8556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:14:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:14:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:14:17.914741 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35717
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35717
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:14:18.023531 8569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:14:18.023823 8569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:14:18.023865 8569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:14:18.027797 8569 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:14:18.027863 8569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:14:18.027885 8569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:14:18.027908 8569 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:14:18.027930 8569 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:14:18.033221 8569 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35717
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:14:18.034219 8569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:14:18.035183 8569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:14:18.043188 8574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:18.043314 8569 server_base.cc:1034] running on GCE node
W20260924 04:14:18.043268 8575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:18.043253 8577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:18.043995 8569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:14:18.045070 8569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:14:18.046242 8569 hybrid_clock.cc:648] HybridClock initialized: now 1790223258046222 us; error 31 us; skew 500 ppm
Sep 24 04:14:18 dist-test-slave-p401 krb5kdc[8556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:14:18.055832 8569 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:14:18.056825 8569 webserver.cc:460] Webserver started at http://127.0.98.62:43591/ using document root <none> and password file <none>
I20260924 04:14:18.057394 8569 fs_manager.cc:359] Metadata directory not provided
I20260924 04:14:18.057461 8569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:14:18.057677 8569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:14:18.059894 8569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/master-0/data/instance:
uuid: "b0ea51b6b331408f9e301819cbf1f6bb"
format_stamp: "Formatted at 2026-09-24 04:14:18 on dist-test-slave-p401"
server_key: "b07a9fc56489dbaae8c88435529c9078"
server_key_iv: "273d754e5029c83059e4c27cb7a55ee5"
server_key_version: "encryptionkey@0"
I20260924 04:14:18.060485 8569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/master-0/wal/instance:
uuid: "b0ea51b6b331408f9e301819cbf1f6bb"
format_stamp: "Formatted at 2026-09-24 04:14:18 on dist-test-slave-p401"
server_key: "b07a9fc56489dbaae8c88435529c9078"
server_key_iv: "273d754e5029c83059e4c27cb7a55ee5"
server_key_version: "encryptionkey@0"
I20260924 04:14:18.064038 8569 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 04:14:18.066249 8584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:18.067426 8569 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:14:18.067540 8569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/master-0/wal
uuid: "b0ea51b6b331408f9e301819cbf1f6bb"
format_stamp: "Formatted at 2026-09-24 04:14:18 on dist-test-slave-p401"
server_key: "b07a9fc56489dbaae8c88435529c9078"
server_key_iv: "273d754e5029c83059e4c27cb7a55ee5"
server_key_version: "encryptionkey@0"
I20260924 04:14:18.067628 8569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:14:18.077865 8569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:14:18.080356 8569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:14:18.080514 8569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:14:18.086836 8569 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35717
I20260924 04:14:18.086853 8636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35717 every 8 connection(s)
I20260924 04:14:18.087733 8569 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:14:18.090262 8637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:14:18.090703 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 8569
I20260924 04:14:18.090818 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/master-0/wal/instance
I20260924 04:14:18.091109 392 external_mini_cluster.cc:1428] Setting key 9a50b5ef4ea3f180c2e2ae1f78b6ba52
I20260924 04:14:18.096608 8637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0ea51b6b331408f9e301819cbf1f6bb: Bootstrap starting.
I20260924 04:14:18.099015 8637 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0ea51b6b331408f9e301819cbf1f6bb: Neither blocks nor log segments found. Creating new log.
I20260924 04:14:18.099735 8637 log.cc:824] T 00000000000000000000000000000000 P b0ea51b6b331408f9e301819cbf1f6bb: Log is configured to *not* fsync() on all Append() calls
Sep 24 04:14:18 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:14:18.101773 8637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0ea51b6b331408f9e301819cbf1f6bb: No bootstrap required, opened a new log
I20260924 04:14:18.105625 8640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:18.092541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58582 (local address 127.0.98.62:35717)
0924 04:14:18.092945 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:18.092962 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:18.093006 (+ 44us) server_negotiation.cc:407] Connection header received
0924 04:14:18.093783 (+ 777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:18.093805 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:18.094202 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:18.094555 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:18.095208 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:18.096659 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:18.097479 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:18.098131 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:18.098277 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:18.100830 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:18.100836 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:18.100851 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:18.100880 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:18.102689 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:18.103086 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:18.103092 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:18.103098 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:18.103253 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:18.103491 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:18.103494 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:18.103496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:18.104008 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:18.104195 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:18.104405 (+ 210us) server_negotiation.cc:299] Negotiation successful
0924 04:14:18.104690 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":127,"threads_started":1}
I20260924 04:14:18.106004 8637 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0ea51b6b331408f9e301819cbf1f6bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ea51b6b331408f9e301819cbf1f6bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35717 } }
I20260924 04:14:18.106259 8637 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0ea51b6b331408f9e301819cbf1f6bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:14:18.106321 8637 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0ea51b6b331408f9e301819cbf1f6bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0ea51b6b331408f9e301819cbf1f6bb, State: Initialized, Role: FOLLOWER
I20260924 04:14:18.106783 8637 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0ea51b6b331408f9e301819cbf1f6bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ea51b6b331408f9e301819cbf1f6bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35717 } }
I20260924 04:14:18.106921 8637 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0ea51b6b331408f9e301819cbf1f6bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:14:18.106986 8637 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0ea51b6b331408f9e301819cbf1f6bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:14:18.107079 8637 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0ea51b6b331408f9e301819cbf1f6bb [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:14:18.108063 8637 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0ea51b6b331408f9e301819cbf1f6bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ea51b6b331408f9e301819cbf1f6bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35717 } }
I20260924 04:14:18.108361 8637 leader_election.cc:302] T 00000000000000000000000000000000 P b0ea51b6b331408f9e301819cbf1f6bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0ea51b6b331408f9e301819cbf1f6bb; no voters:
I20260924 04:14:18.108628 8637 leader_election.cc:288] T 00000000000000000000000000000000 P b0ea51b6b331408f9e301819cbf1f6bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:14:18.108778 8642 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0ea51b6b331408f9e301819cbf1f6bb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:14:18.108963 8642 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0ea51b6b331408f9e301819cbf1f6bb [term 1 LEADER]: Becoming Leader. State: Replica: b0ea51b6b331408f9e301819cbf1f6bb, State: Running, Role: LEADER
I20260924 04:14:18.109233 8642 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0ea51b6b331408f9e301819cbf1f6bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ea51b6b331408f9e301819cbf1f6bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35717 } }
I20260924 04:14:18.109709 8637 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0ea51b6b331408f9e301819cbf1f6bb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:14:18.110678 8643 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0ea51b6b331408f9e301819cbf1f6bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0ea51b6b331408f9e301819cbf1f6bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ea51b6b331408f9e301819cbf1f6bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35717 } } }
I20260924 04:14:18.110793 8643 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0ea51b6b331408f9e301819cbf1f6bb [sys.catalog]: This master's current role is: LEADER
I20260924 04:14:18.110906 8644 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0ea51b6b331408f9e301819cbf1f6bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0ea51b6b331408f9e301819cbf1f6bb. Latest consensus state: current_term: 1 leader_uuid: "b0ea51b6b331408f9e301819cbf1f6bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ea51b6b331408f9e301819cbf1f6bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35717 } } }
I20260924 04:14:18.111052 8644 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0ea51b6b331408f9e301819cbf1f6bb [sys.catalog]: This master's current role is: LEADER
I20260924 04:14:18.111367 8651 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:14:18.114181 8651 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:14:18.127292 8651 catalog_manager.cc:1347] Generated new cluster ID: 9e2f32a6cf9545a69e20d502bbe6855b
I20260924 04:14:18.127381 8651 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:14:18.153497 8651 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:14:18.154486 8651 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:14:18.162405 8651 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0ea51b6b331408f9e301819cbf1f6bb: Generated new TSK 0
I20260924 04:14:18.163311 8651 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:14:18.193917 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35717
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:14:18.302744 8663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:14:18.303021 8663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:14:18.303066 8663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:14:18.307129 8663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:14:18.307199 8663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:14:18.307293 8663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:14:18.312882 8663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:14:18.313949 8663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:14:18.314967 8663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:14:18.324003 8669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:18.324009 8671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:18.324021 8668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:18.324179 8663 server_base.cc:1034] running on GCE node
I20260924 04:14:18.324599 8663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:14:18.325196 8663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:14:18.326350 8663 hybrid_clock.cc:648] HybridClock initialized: now 1790223258326333 us; error 33 us; skew 500 ppm
Sep 24 04:14:18 dist-test-slave-p401 krb5kdc[8556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:14:18.328908 8663 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:14:18.329703 8663 webserver.cc:460] Webserver started at http://127.0.98.1:43273/ using document root <none> and password file <none>
I20260924 04:14:18.330204 8663 fs_manager.cc:359] Metadata directory not provided
I20260924 04:14:18.330256 8663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:14:18.330420 8663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:14:18.332567 8663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-0/data/instance:
uuid: "6d7cfba9fe0c4068bb75fedb9be03d42"
format_stamp: "Formatted at 2026-09-24 04:14:18 on dist-test-slave-p401"
server_key: "79c19e8ded2a29491322a8f6b8294eea"
server_key_iv: "644be9e670dae7afb5318a1fd7b13f73"
server_key_version: "encryptionkey@0"
I20260924 04:14:18.333057 8663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6d7cfba9fe0c4068bb75fedb9be03d42"
format_stamp: "Formatted at 2026-09-24 04:14:18 on dist-test-slave-p401"
server_key: "79c19e8ded2a29491322a8f6b8294eea"
server_key_iv: "644be9e670dae7afb5318a1fd7b13f73"
server_key_version: "encryptionkey@0"
I20260924 04:14:18.336428 8663 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:14:18.338608 8678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:18.339802 8663 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:14:18.339896 8663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-0/wal
uuid: "6d7cfba9fe0c4068bb75fedb9be03d42"
format_stamp: "Formatted at 2026-09-24 04:14:18 on dist-test-slave-p401"
server_key: "79c19e8ded2a29491322a8f6b8294eea"
server_key_iv: "644be9e670dae7afb5318a1fd7b13f73"
server_key_version: "encryptionkey@0"
I20260924 04:14:18.339972 8663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:14:18.352916 8663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:14:18.355687 8663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:14:18.355832 8663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:14:18.356316 8663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:14:18.357062 8663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:14:18.357105 8663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:18.357143 8663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:14:18.357162 8663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:18.364977 8663 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40025
I20260924 04:14:18.365008 8791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40025 every 8 connection(s)
I20260924 04:14:18.365819 8663 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:14:18.369848 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 8663
I20260924 04:14:18.369975 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:14:18.370246 392 external_mini_cluster.cc:1428] Setting key 53ebb4a7c7000363390882dc920364c0
Sep 24 04:14:18 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:14:18.379666 8640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:18.367534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60803 (local address 127.0.98.62:35717)
0924 04:14:18.367680 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:18.367685 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:18.368490 (+ 805us) server_negotiation.cc:407] Connection header received
0924 04:14:18.369819 (+ 1329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:18.369823 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:18.369879 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:18.369971 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:18.371291 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:18.371907 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:18.373468 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:18.373800 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:18.373911 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:18.376429 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:18.376433 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:18.376435 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:18.376458 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:18.377722 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:18.378192 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:18.378195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:18.378196 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:18.378269 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:18.378577 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:18.378580 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:18.378582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:18.378755 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:18.378843 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:18.379377 (+ 534us) server_negotiation.cc:299] Negotiation successful
0924 04:14:18.379504 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 04:14:18.380589 8794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:18.367894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35717 (local address 127.0.98.1:60803)
0924 04:14:18.368338 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:18.368381 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:18.369589 (+ 1208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:18.370162 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:18.370168 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:18.370562 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:18.371146 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:18.371160 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:18.372041 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:18.372045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:18.373350 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:18.373361 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:18.373942 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:18.373945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:18.374090 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:18.374643 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:18.374662 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:18.376259 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:14:18.377829 (+ 1570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:18.377835 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:18.377852 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:18.378099 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:18.378354 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:18.378357 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:18.378359 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:18.378496 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:18.378848 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:14:18.378856 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:14:18.379128 (+ 272us) client_negotiation.cc:769] Sending connection context
0924 04:14:18.379332 (+ 204us) client_negotiation.cc:240] Negotiation successful
0924 04:14:18.379665 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":110,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:14:18.381951 8792 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35717
I20260924 04:14:18.382189 8792 heartbeater.cc:464] Registering TS with master...
I20260924 04:14:18.382681 8792 heartbeater.cc:511] Master 127.0.98.62:35717 requested a full tablet report, sending...
I20260924 04:14:18.384151 8601 ts_manager.cc:194] Registered new tserver with Master: 6d7cfba9fe0c4068bb75fedb9be03d42 (127.0.98.1:40025)
I20260924 04:14:18.385303 8601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60803
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:14:18.397948 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35717
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:14:18.505107 8797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:14:18.505366 8797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:14:18.505425 8797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:14:18.509493 8797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:14:18.509559 8797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:14:18.509634 8797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:14:18.515029 8797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:14:18.516094 8797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:14:18.517102 8797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:14:18.526260 8802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:18.526275 8805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:18.526324 8803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:18.526577 8797 server_base.cc:1034] running on GCE node
I20260924 04:14:18.527004 8797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:14:18.527642 8797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:14:18.528796 8797 hybrid_clock.cc:648] HybridClock initialized: now 1790223258528767 us; error 43 us; skew 500 ppm
Sep 24 04:14:18 dist-test-slave-p401 krb5kdc[8556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:14:18.531421 8797 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:14:18.532429 8797 webserver.cc:460] Webserver started at http://127.0.98.2:37489/ using document root <none> and password file <none>
I20260924 04:14:18.533022 8797 fs_manager.cc:359] Metadata directory not provided
I20260924 04:14:18.533087 8797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:14:18.533280 8797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:14:18.535282 8797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-1/data/instance:
uuid: "089725185d2e46f8a2fca5cbd6e75077"
format_stamp: "Formatted at 2026-09-24 04:14:18 on dist-test-slave-p401"
server_key: "808ec72ac8135634417c6540142d9778"
server_key_iv: "13ca34ab9542c6954420480576a9e62b"
server_key_version: "encryptionkey@0"
I20260924 04:14:18.535846 8797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-1/wal/instance:
uuid: "089725185d2e46f8a2fca5cbd6e75077"
format_stamp: "Formatted at 2026-09-24 04:14:18 on dist-test-slave-p401"
server_key: "808ec72ac8135634417c6540142d9778"
server_key_iv: "13ca34ab9542c6954420480576a9e62b"
server_key_version: "encryptionkey@0"
I20260924 04:14:18.539460 8797 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20260924 04:14:18.541792 8812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:18.543002 8797 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:14:18.543103 8797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-1/wal
uuid: "089725185d2e46f8a2fca5cbd6e75077"
format_stamp: "Formatted at 2026-09-24 04:14:18 on dist-test-slave-p401"
server_key: "808ec72ac8135634417c6540142d9778"
server_key_iv: "13ca34ab9542c6954420480576a9e62b"
server_key_version: "encryptionkey@0"
I20260924 04:14:18.543200 8797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:14:18.555413 8797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:14:18.558009 8797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:14:18.558179 8797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:14:18.558734 8797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:14:18.559535 8797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:14:18.559581 8797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:18.559633 8797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:14:18.559667 8797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:18.567571 8797 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46397
I20260924 04:14:18.567598 8925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46397 every 8 connection(s)
I20260924 04:14:18.568444 8797 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:14:18.573731 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 8797
I20260924 04:14:18.573848 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:14:18.574091 392 external_mini_cluster.cc:1428] Setting key aaa4ed00e2397c1e6b564f6a3e07bd52
Sep 24 04:14:18 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:14:18.581882 8640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:18.570084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56319 (local address 127.0.98.62:35717)
0924 04:14:18.570208 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:18.570212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:18.571006 (+ 794us) server_negotiation.cc:407] Connection header received
0924 04:14:18.572449 (+ 1443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:18.572452 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:18.572497 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:18.572559 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:18.573721 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:18.574375 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:18.575827 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:18.576136 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:18.576243 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:18.578657 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:18.578660 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:18.578662 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:18.578685 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:18.580034 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:18.580503 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:18.580507 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:18.580509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:18.580581 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:18.580871 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:18.580874 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:18.580876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:18.581046 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:18.581127 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:18.581601 (+ 474us) server_negotiation.cc:299] Negotiation successful
0924 04:14:18.581724 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 04:14:18.582690 8928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:18.570456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35717 (local address 127.0.98.2:56319)
0924 04:14:18.570866 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:18.570907 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:18.572244 (+ 1337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:18.572710 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:18.572716 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:18.573069 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:18.573562 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:18.573581 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:18.574500 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:18.574503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:18.575712 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:18.575721 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:18.576272 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:18.576275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:18.576453 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:18.577042 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:18.577061 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:18.578498 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:14:18.580141 (+ 1643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:18.580146 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:18.580169 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:18.580413 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:18.580656 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:18.580659 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:18.580661 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:18.580789 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:18.581136 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:14:18.581143 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:14:18.581384 (+ 241us) client_negotiation.cc:769] Sending connection context
0924 04:14:18.581545 (+ 161us) client_negotiation.cc:240] Negotiation successful
0924 04:14:18.581856 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":101,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:14:18.583892 8926 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35717
I20260924 04:14:18.584110 8926 heartbeater.cc:464] Registering TS with master...
I20260924 04:14:18.584570 8926 heartbeater.cc:511] Master 127.0.98.62:35717 requested a full tablet report, sending...
I20260924 04:14:18.585588 8601 ts_manager.cc:194] Registered new tserver with Master: 089725185d2e46f8a2fca5cbd6e75077 (127.0.98.2:46397)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 04:14:18.586154 8601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56319
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:14:18.602417 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35717
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:14:18.707894 8931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:14:18.708146 8931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:14:18.708187 8931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:14:18.712265 8931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:14:18.712317 8931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:14:18.712373 8931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:14:18.717499 8931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:14:18.718436 8931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:14:18.719389 8931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:14:18.730571 8936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:18.730566 8937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:18.730903 8939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:18.730942 8931 server_base.cc:1034] running on GCE node
I20260924 04:14:18.731379 8931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:14:18.732002 8931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:14:18.733153 8931 hybrid_clock.cc:648] HybridClock initialized: now 1790223258733131 us; error 35 us; skew 500 ppm
Sep 24 04:14:18 dist-test-slave-p401 krb5kdc[8556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:14:18.735698 8931 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:14:18.736547 8931 webserver.cc:460] Webserver started at http://127.0.98.3:42493/ using document root <none> and password file <none>
I20260924 04:14:18.737071 8931 fs_manager.cc:359] Metadata directory not provided
I20260924 04:14:18.737131 8931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:14:18.737320 8931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:14:18.739311 8931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-2/data/instance:
uuid: "61cfc63e79e5476cbebb84c3d19bc0a5"
format_stamp: "Formatted at 2026-09-24 04:14:18 on dist-test-slave-p401"
server_key: "77d0b7b0d1fb67de008fc5315e8f911c"
server_key_iv: "e1f09f6f2902c43735b251e42426958e"
server_key_version: "encryptionkey@0"
I20260924 04:14:18.739869 8931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-2/wal/instance:
uuid: "61cfc63e79e5476cbebb84c3d19bc0a5"
format_stamp: "Formatted at 2026-09-24 04:14:18 on dist-test-slave-p401"
server_key: "77d0b7b0d1fb67de008fc5315e8f911c"
server_key_iv: "e1f09f6f2902c43735b251e42426958e"
server_key_version: "encryptionkey@0"
I20260924 04:14:18.743379 8931 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260924 04:14:18.745756 8946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:18.746973 8931 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:14:18.747079 8931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-2/wal
uuid: "61cfc63e79e5476cbebb84c3d19bc0a5"
format_stamp: "Formatted at 2026-09-24 04:14:18 on dist-test-slave-p401"
server_key: "77d0b7b0d1fb67de008fc5315e8f911c"
server_key_iv: "e1f09f6f2902c43735b251e42426958e"
server_key_version: "encryptionkey@0"
I20260924 04:14:18.747175 8931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:14:18.772284 8931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:14:18.775319 8931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:14:18.775499 8931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:14:18.776095 8931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:14:18.776921 8931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:14:18.776966 8931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:18.777020 8931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:14:18.777047 8931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:18.785564 8931 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33339
I20260924 04:14:18.785593 9059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33339 every 8 connection(s)
I20260924 04:14:18.786473 8931 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:14:18.788170 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 8931
I20260924 04:14:18.788287 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:14:18.788579 392 external_mini_cluster.cc:1428] Setting key 5dfa9d9afbd14df42aa5ef1b74a5bb36
Sep 24 04:14:18 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:14:18.799297 8640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:18.788467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34425 (local address 127.0.98.62:35717)
0924 04:14:18.788606 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:18.788611 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:18.789576 (+ 965us) server_negotiation.cc:407] Connection header received
0924 04:14:18.790945 (+ 1369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:18.790948 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:18.790993 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:18.791060 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:18.792230 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:18.792782 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:18.793848 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:18.794088 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:18.794184 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:18.796543 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:18.796546 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:18.796548 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:18.796572 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:18.797643 (+ 1071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:18.798080 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:18.798082 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:18.798083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:18.798149 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:18.798403 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:18.798405 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:18.798406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:18.798561 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:18.798625 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:18.799051 (+ 426us) server_negotiation.cc:299] Negotiation successful
0924 04:14:18.799141 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 04:14:18.800223 9062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:18.788931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35717 (local address 127.0.98.3:34425)
0924 04:14:18.789426 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:18.789480 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:18.790746 (+ 1266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:18.791196 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:18.791203 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:18.791599 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:18.792092 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:18.792105 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:18.792877 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:18.792879 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:18.793742 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:18.793751 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:18.794220 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:18.794223 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:18.794371 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:18.794884 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:18.794900 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:18.796392 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:14:18.797741 (+ 1349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:18.797746 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:18.797758 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:18.798005 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:18.798210 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:18.798212 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:18.798214 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:18.798341 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:18.798625 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:14:18.798630 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:14:18.798871 (+ 241us) client_negotiation.cc:769] Sending connection context
0924 04:14:18.799023 (+ 152us) client_negotiation.cc:240] Negotiation successful
0924 04:14:18.799329 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":118,"threads_started":1}
I20260924 04:14:18.801391 9060 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35717
I20260924 04:14:18.801615 9060 heartbeater.cc:464] Registering TS with master...
I20260924 04:14:18.802151 9060 heartbeater.cc:511] Master 127.0.98.62:35717 requested a full tablet report, sending...
I20260924 04:14:18.803325 8601 ts_manager.cc:194] Registered new tserver with Master: 61cfc63e79e5476cbebb84c3d19bc0a5 (127.0.98.3:33339)
I20260924 04:14:18.803964 8601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34425
I20260924 04:14:18.814075 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260924 04:14:18.879681 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 8663
I20260924 04:14:18.884148 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 8797
I20260924 04:14:18.888612 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 8931
I20260924 04:14:18.892675 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 8569
I20260924 04:14:18.898216 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35717
--webserver_interface=127.0.98.62
--webserver_port=43591
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35717
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:14:19.007661 9069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:14:19.007956 9069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:14:19.007997 9069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:14:19.012042 9069 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:14:19.012115 9069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:14:19.012135 9069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:14:19.012149 9069 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:14:19.012164 9069 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:14:19.017607 9069 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35717
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=43591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:14:19.018694 9069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:14:19.019784 9069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:14:19.028142 9074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:19.028191 9069 server_base.cc:1034] running on GCE node
W20260924 04:14:19.028193 9075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:19.028142 9077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:19.028928 9069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:14:19.030099 9069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:14:19.031322 9069 hybrid_clock.cc:648] HybridClock initialized: now 1790223259031297 us; error 42 us; skew 500 ppm
Sep 24 04:14:19 dist-test-slave-p401 krb5kdc[8556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223259, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:14:19.034102 9069 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:14:19.035018 9069 webserver.cc:460] Webserver started at http://127.0.98.62:43591/ using document root <none> and password file <none>
I20260924 04:14:19.035624 9069 fs_manager.cc:359] Metadata directory not provided
I20260924 04:14:19.035706 9069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:14:19.039919 9069 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:14:19.042091 9084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:19.043422 9069 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260924 04:14:19.043534 9069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/master-0/wal
uuid: "b0ea51b6b331408f9e301819cbf1f6bb"
format_stamp: "Formatted at 2026-09-24 04:14:18 on dist-test-slave-p401"
server_key: "b07a9fc56489dbaae8c88435529c9078"
server_key_iv: "273d754e5029c83059e4c27cb7a55ee5"
server_key_version: "encryptionkey@0"
I20260924 04:14:19.044062 9069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:14:19.053926 9069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:14:19.056816 9069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:14:19.056984 9069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:14:19.063699 9136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35717 every 8 connection(s)
I20260924 04:14:19.063705 9069 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35717
I20260924 04:14:19.064776 9069 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:14:19.069212 9137 sys_catalog.cc:263] Verifying existing consensus state
I20260924 04:14:19.070909 9137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0ea51b6b331408f9e301819cbf1f6bb: Bootstrap starting.
I20260924 04:14:19.074723 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 9069
I20260924 04:14:19.075322 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:40025
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:35717
--webserver_port=43273
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:14:19.088439 9137 log.cc:824] T 00000000000000000000000000000000 P b0ea51b6b331408f9e301819cbf1f6bb: Log is configured to *not* fsync() on all Append() calls
I20260924 04:14:19.095839 9137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0ea51b6b331408f9e301819cbf1f6bb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:14:19.096242 9137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0ea51b6b331408f9e301819cbf1f6bb: Bootstrap complete.
I20260924 04:14:19.100540 9137 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0ea51b6b331408f9e301819cbf1f6bb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ea51b6b331408f9e301819cbf1f6bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35717 } }
I20260924 04:14:19.101073 9137 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0ea51b6b331408f9e301819cbf1f6bb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0ea51b6b331408f9e301819cbf1f6bb, State: Initialized, Role: FOLLOWER
I20260924 04:14:19.101578 9137 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0ea51b6b331408f9e301819cbf1f6bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ea51b6b331408f9e301819cbf1f6bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35717 } }
I20260924 04:14:19.101739 9137 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0ea51b6b331408f9e301819cbf1f6bb [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:14:19.101814 9137 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0ea51b6b331408f9e301819cbf1f6bb [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:14:19.101908 9137 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0ea51b6b331408f9e301819cbf1f6bb [term 1 FOLLOWER]: Advancing to term 2
I20260924 04:14:19.103953 9137 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0ea51b6b331408f9e301819cbf1f6bb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ea51b6b331408f9e301819cbf1f6bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35717 } }
I20260924 04:14:19.104282 9137 leader_election.cc:302] T 00000000000000000000000000000000 P b0ea51b6b331408f9e301819cbf1f6bb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0ea51b6b331408f9e301819cbf1f6bb; no voters:
I20260924 04:14:19.104575 9137 leader_election.cc:288] T 00000000000000000000000000000000 P b0ea51b6b331408f9e301819cbf1f6bb [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 04:14:19.104746 9141 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0ea51b6b331408f9e301819cbf1f6bb [term 2 FOLLOWER]: Leader election won for term 2
I20260924 04:14:19.104947 9141 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0ea51b6b331408f9e301819cbf1f6bb [term 2 LEADER]: Becoming Leader. State: Replica: b0ea51b6b331408f9e301819cbf1f6bb, State: Running, Role: LEADER
I20260924 04:14:19.105289 9141 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0ea51b6b331408f9e301819cbf1f6bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ea51b6b331408f9e301819cbf1f6bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35717 } }
I20260924 04:14:19.105881 9137 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0ea51b6b331408f9e301819cbf1f6bb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:14:19.106694 9142 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0ea51b6b331408f9e301819cbf1f6bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b0ea51b6b331408f9e301819cbf1f6bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ea51b6b331408f9e301819cbf1f6bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35717 } } }
I20260924 04:14:19.106827 9142 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0ea51b6b331408f9e301819cbf1f6bb [sys.catalog]: This master's current role is: LEADER
I20260924 04:14:19.107113 9143 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0ea51b6b331408f9e301819cbf1f6bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0ea51b6b331408f9e301819cbf1f6bb. Latest consensus state: current_term: 2 leader_uuid: "b0ea51b6b331408f9e301819cbf1f6bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ea51b6b331408f9e301819cbf1f6bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35717 } } }
I20260924 04:14:19.107301 9150 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:14:19.109166 9143 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0ea51b6b331408f9e301819cbf1f6bb [sys.catalog]: This master's current role is: LEADER
I20260924 04:14:19.110572 9150 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:14:19.111094 9150 catalog_manager.cc:1259] Loaded cluster ID: 9e2f32a6cf9545a69e20d502bbe6855b
I20260924 04:14:19.111145 9150 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:14:19.112851 9150 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:14:19.113411 9150 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b0ea51b6b331408f9e301819cbf1f6bb: Loaded TSK: 0
I20260924 04:14:19.114347 9150 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 04:14:19.215876 9139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:14:19.216305 9139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:14:19.216378 9139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:14:19.222446 9139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:14:19.222538 9139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:14:19.222628 9139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:14:19.230057 9139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:40025
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43273
--tserver_master_addrs=127.0.98.62:35717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:14:19.231443 9139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:14:19.232722 9139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:14:19.244606 9164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:19.244648 9163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:19.244679 9166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:19.245090 9139 server_base.cc:1034] running on GCE node
I20260924 04:14:19.245555 9139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:14:19.246346 9139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:14:19.247531 9139 hybrid_clock.cc:648] HybridClock initialized: now 1790223259247500 us; error 42 us; skew 500 ppm
Sep 24 04:14:19 dist-test-slave-p401 krb5kdc[8556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223259, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:14:19.250869 9139 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 04:14:19.252058 9139 webserver.cc:460] Webserver started at http://127.0.98.1:43273/ using document root <none> and password file <none>
I20260924 04:14:19.252763 9139 fs_manager.cc:359] Metadata directory not provided
I20260924 04:14:19.252874 9139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:14:19.258019 9139 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:14:19.260710 9173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:19.262264 9139 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 04:14:19.262389 9139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-0/wal
uuid: "6d7cfba9fe0c4068bb75fedb9be03d42"
format_stamp: "Formatted at 2026-09-24 04:14:18 on dist-test-slave-p401"
server_key: "79c19e8ded2a29491322a8f6b8294eea"
server_key_iv: "644be9e670dae7afb5318a1fd7b13f73"
server_key_version: "encryptionkey@0"
I20260924 04:14:19.263095 9139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:14:19.290069 9139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:14:19.293795 9139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:14:19.293984 9139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:14:19.294672 9139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:14:19.295675 9139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:14:19.295732 9139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:19.295783 9139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:14:19.295811 9139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:19.305176 9139 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40025
I20260924 04:14:19.305193 9286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40025 every 8 connection(s)
I20260924 04:14:19.306205 9139 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:14:19.312582 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 9139
I20260924 04:14:19.312999 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:46397
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:35717
--webserver_port=37489
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 04:14:19 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:14:19 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 04:14:19.320031 9289 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:19.308407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51661 (local address 127.0.98.62:35717)
0924 04:14:19.308883 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:19.308898 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:19.309184 (+ 286us) server_negotiation.cc:407] Connection header received
0924 04:14:19.310724 (+ 1540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:19.310737 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:19.311057 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:19.311417 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:19.312644 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:19.313715 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:19.315032 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:19.315590 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:19.315725 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:19.319276 (+ 3551us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51661: BlockingRecv error: recv got EOF from 127.0.98.1:51661 (error 108)
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":121,"threads_started":1}
I20260924 04:14:19.320039 9290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:19.308507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35717 (local address 127.0.98.1:51661)
0924 04:14:19.309028 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:19.309078 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:19.310324 (+ 1246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:19.311514 (+ 1190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:19.311521 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:19.311922 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:19.312478 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:19.312500 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:19.313907 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:19.313911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:19.314891 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:19.314903 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:19.315770 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:19.315774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:19.315916 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:19.316478 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:19.316494 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:19.318974 (+ 2480us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35717: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":116,"threads_started":1}
W20260924 04:14:19.320784 9287 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35717 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:35717: Client connection negotiation failed: client connection to 127.0.98.62:35717: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 24 04:14:19 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:14:19 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:14:19.326854 9290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:19.321299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35717 (local address 127.0.98.1:41893)
0924 04:14:19.321447 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:19.321464 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:19.321554 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:19.321802 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:19.321805 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:19.321981 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:19.322101 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:19.322108 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:19.322866 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:19.322870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:19.323841 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:19.323853 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:19.324314 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:19.324318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:19.324413 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:19.324890 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:19.324904 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:19.326720 (+ 1816us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35717: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":62}
W20260924 04:14:19.327070 9289 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:19.321428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41893 (local address 127.0.98.62:35717)
0924 04:14:19.321535 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:19.321538 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:19.321551 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:14:19.321649 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:19.321652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:19.321696 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:19.321762 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:19.322236 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:19.322759 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:19.323981 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:19.324211 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:19.324299 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:19.326864 (+ 2565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41893: BlockingRecv error: recv got EOF from 127.0.98.1:41893 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 24 04:14:19 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:14:19 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:14:19.331691 9290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:19.327387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35717 (local address 127.0.98.1:55775)
0924 04:14:19.327492 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:19.327506 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:19.327581 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:19.327826 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:19.327828 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:19.327958 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:19.328024 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:19.328029 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:19.328691 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:19.328695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:19.329199 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:19.329206 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:19.329583 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:19.329586 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:19.329655 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:19.330096 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:19.330105 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:19.331572 (+ 1467us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35717: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 04:14:19.331799 9289 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:19.327446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55775 (local address 127.0.98.62:35717)
0924 04:14:19.327538 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:19.327541 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:19.327553 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:14:19.327695 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:19.327698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:19.327738 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:19.327798 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:19.328118 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:19.328601 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:19.329293 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:19.329498 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:19.329575 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:19.331705 (+ 2130us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55775: BlockingRecv error: recv got EOF from 127.0.98.1:55775 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260924 04:14:19.331898 9287 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 04:14:19.424525 9291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:14:19.424827 9291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:14:19.424890 9291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:14:19.428998 9291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:14:19.429078 9291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:14:19.429167 9291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:14:19.434650 9291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:46397
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37489
--tserver_master_addrs=127.0.98.62:35717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:14:19.435774 9291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:14:19.436786 9291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:14:19.446557 9291 server_base.cc:1034] running on GCE node
W20260924 04:14:19.446587 9296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:19.446502 9297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:19.446502 9299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:19.447178 9291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:14:19.447945 9291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:14:19.449138 9291 hybrid_clock.cc:648] HybridClock initialized: now 1790223259449113 us; error 34 us; skew 500 ppm
Sep 24 04:14:19 dist-test-slave-p401 krb5kdc[8556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223259, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:14:19.451936 9291 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 04:14:19.452872 9291 webserver.cc:460] Webserver started at http://127.0.98.2:37489/ using document root <none> and password file <none>
I20260924 04:14:19.453416 9291 fs_manager.cc:359] Metadata directory not provided
I20260924 04:14:19.453496 9291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:14:19.457435 9291 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260924 04:14:19.459582 9306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:19.460887 9291 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260924 04:14:19.461018 9291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-1/wal
uuid: "089725185d2e46f8a2fca5cbd6e75077"
format_stamp: "Formatted at 2026-09-24 04:14:18 on dist-test-slave-p401"
server_key: "808ec72ac8135634417c6540142d9778"
server_key_iv: "13ca34ab9542c6954420480576a9e62b"
server_key_version: "encryptionkey@0"
I20260924 04:14:19.461583 9291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:14:19.481040 9291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:14:19.483987 9291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:14:19.484165 9291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:14:19.484758 9291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:14:19.485584 9291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:14:19.485639 9291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:19.485692 9291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:14:19.485718 9291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:19.494115 9291 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46397
I20260924 04:14:19.494143 9419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46397 every 8 connection(s)
I20260924 04:14:19.495144 9291 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:14:19.500087 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 9291
I20260924 04:14:19.500571 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33339
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:35717
--webserver_port=42493
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 04:14:19 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:14:19 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 04:14:19.507359 9289 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:19.497037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42267 (local address 127.0.98.62:35717)
0924 04:14:19.497190 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:19.497194 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:19.498258 (+ 1064us) server_negotiation.cc:407] Connection header received
0924 04:14:19.499593 (+ 1335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:19.499596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:19.499653 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:19.499731 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:19.501304 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:19.501929 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:19.503160 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:19.503406 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:19.503485 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:19.507179 (+ 3694us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42267: BlockingRecv error: recv got EOF from 127.0.98.2:42267 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 04:14:19.507964 9422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:19.497636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35717 (local address 127.0.98.2:42267)
0924 04:14:19.498101 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:19.498146 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:19.499384 (+ 1238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:19.499940 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:19.499950 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:19.500472 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:19.501151 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:19.501169 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:19.502065 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:19.502068 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:19.503027 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:19.503037 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:19.503547 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:19.503551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:19.503735 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:19.504391 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:19.504413 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:19.507013 (+ 2600us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35717: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":122,"threads_started":1}
W20260924 04:14:19.508546 9420 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35717 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:35717: Client connection negotiation failed: client connection to 127.0.98.62:35717: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 24 04:14:19 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:14:19 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:14:19.513811 9422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:19.509044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35717 (local address 127.0.98.2:47129)
0924 04:14:19.509203 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:19.509217 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:19.509294 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:19.509525 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:19.509528 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:19.509707 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:19.509798 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:19.509805 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:19.510532 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:19.510535 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:19.511088 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:19.511096 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:19.511573 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:19.511576 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:19.511669 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:19.512093 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:19.512104 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:19.513692 (+ 1588us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35717: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97}
W20260924 04:14:19.513916 9289 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:19.509137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47129 (local address 127.0.98.62:35717)
0924 04:14:19.509250 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:19.509254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:19.509267 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:14:19.509393 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:19.509396 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:19.509439 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:19.509507 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:19.509905 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:19.510424 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:19.511190 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:19.511476 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:19.511563 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:19.513818 (+ 2255us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:47129: BlockingRecv error: recv got EOF from 127.0.98.2:47129 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 24 04:14:19 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:14:19 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:14:19.519196 9422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:19.514746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35717 (local address 127.0.98.2:37321)
0924 04:14:19.514905 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:19.514918 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:19.514988 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:19.515217 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:19.515219 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:19.515378 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:19.515445 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:19.515449 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:19.516120 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:19.516123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:19.516642 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:19.516649 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:19.517023 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:19.517027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:19.517097 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:19.517511 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:19.517525 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:19.519071 (+ 1546us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35717: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96}
W20260924 04:14:19.519322 9289 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:19.514838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37321 (local address 127.0.98.62:35717)
0924 04:14:19.514945 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:19.514949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:19.514961 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:14:19.515067 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:19.515069 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:19.515109 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:19.515191 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:19.515539 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:19.516026 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:19.516736 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:19.516938 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:19.517016 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:19.519192 (+ 2176us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37321: BlockingRecv error: recv got EOF from 127.0.98.2:37321 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260924 04:14:19.519414 9420 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 04:14:19.609163 9423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:14:19.609445 9423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:14:19.609508 9423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:14:19.613591 9423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:14:19.613660 9423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:14:19.613741 9423 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:14:19.619149 9423 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33339
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42493
--tserver_master_addrs=127.0.98.62:35717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:14:19.620225 9423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:14:19.621220 9423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:14:19.630240 9429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:19.630332 9423 server_base.cc:1034] running on GCE node
W20260924 04:14:19.630240 9431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:19.630234 9428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:19.630802 9423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:14:19.631479 9423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:14:19.632622 9423 hybrid_clock.cc:648] HybridClock initialized: now 1790223259632602 us; error 35 us; skew 500 ppm
Sep 24 04:14:19 dist-test-slave-p401 krb5kdc[8556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223259, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:14:19.635057 9423 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 04:14:19.635896 9423 webserver.cc:460] Webserver started at http://127.0.98.3:42493/ using document root <none> and password file <none>
I20260924 04:14:19.636435 9423 fs_manager.cc:359] Metadata directory not provided
I20260924 04:14:19.636500 9423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:14:19.640148 9423 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260924 04:14:19.642138 9438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:19.643414 9423 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:14:19.643505 9423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-2/wal
uuid: "61cfc63e79e5476cbebb84c3d19bc0a5"
format_stamp: "Formatted at 2026-09-24 04:14:18 on dist-test-slave-p401"
server_key: "77d0b7b0d1fb67de008fc5315e8f911c"
server_key_iv: "e1f09f6f2902c43735b251e42426958e"
server_key_version: "encryptionkey@0"
I20260924 04:14:19.644021 9423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:14:19.664966 9423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:14:19.668010 9423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:14:19.668172 9423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:14:19.668768 9423 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:14:19.669582 9423 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:14:19.669627 9423 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:19.669669 9423 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:14:19.669697 9423 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:19.678078 9423 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33339
I20260924 04:14:19.678107 9551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33339 every 8 connection(s)
I20260924 04:14:19.679050 9423 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:14:19.687119 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 9423
Sep 24 04:14:19 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:14:19 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 04:14:19.689554 9289 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:19.680721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44811 (local address 127.0.98.62:35717)
0924 04:14:19.680838 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:19.680841 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:19.681610 (+ 769us) server_negotiation.cc:407] Connection header received
0924 04:14:19.682903 (+ 1293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:19.682906 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:19.682964 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:19.683035 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:19.684190 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:19.684705 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:19.685807 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:19.686015 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:19.686100 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:19.689439 (+ 3339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44811: BlockingRecv error: recv got EOF from 127.0.98.3:44811 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 04:14:19.690136 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:19.681056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35717 (local address 127.0.98.3:44811)
0924 04:14:19.681461 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:19.681501 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:19.682692 (+ 1191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:19.683223 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:19.683255 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:19.683588 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:19.684048 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:19.684062 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:19.684818 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:19.684820 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:19.685697 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:19.685706 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:19.686145 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:19.686149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:19.686326 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:19.686817 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:19.686832 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:19.689324 (+ 2492us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35717: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":100,"threads_started":1}
W20260924 04:14:19.690618 9552 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35717 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:35717: Client connection negotiation failed: client connection to 127.0.98.62:35717: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 24 04:14:19 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:14:19 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:14:19.696346 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:19.691001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35717 (local address 127.0.98.3:57921)
0924 04:14:19.691095 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:19.691108 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:19.691175 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:19.691394 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:19.691397 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:19.691543 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:19.691642 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:19.691649 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:19.692576 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:19.692579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:19.693340 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:19.693351 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:19.693769 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:19.693772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:19.693865 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:19.694255 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:19.694267 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:19.696211 (+ 1944us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35717: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 04:14:19.696455 9289 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:19.691068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57921 (local address 127.0.98.62:35717)
0924 04:14:19.691164 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:19.691167 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:19.691179 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:14:19.691260 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:19.691263 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:19.691300 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:19.691377 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:19.691755 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:19.692481 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:19.693465 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:19.693669 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:19.693756 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:19.696343 (+ 2587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:57921: BlockingRecv error: recv got EOF from 127.0.98.3:57921 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 04:14:19.697144 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:19.688064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58592 (local address 127.0.98.62:35717)
0924 04:14:19.688303 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:19.688307 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:19.688339 (+ 32us) server_negotiation.cc:407] Connection header received
0924 04:14:19.688374 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:19.688376 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:19.688413 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:19.688489 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:19.689186 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:19.689738 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:19.690709 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:19.690986 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:19.691072 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:19.692685 (+ 1613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:19.692692 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:19.692711 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:19.692741 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:19.694633 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:19.695120 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:19.695127 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:19.695132 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:19.695321 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:19.695652 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:19.695655 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:19.695656 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:19.696077 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:19.696266 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:19.696754 (+ 488us) server_negotiation.cc:299] Negotiation successful
0924 04:14:19.696997 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":82,"threads_started":1}
Sep 24 04:14:19 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:14:19 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:14:19.701476 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:19.697068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35717 (local address 127.0.98.3:54823)
0924 04:14:19.697218 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:19.697232 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:19.697300 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:19.697500 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:19.697502 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:19.697643 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:19.697716 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:19.697720 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:19.698376 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:19.698379 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:19.698906 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:19.698914 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:19.699462 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:19.699465 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:19.699538 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:19.699918 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:19.699929 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:19.701391 (+ 1462us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35717: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":90}
W20260924 04:14:19.701611 9556 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:19.697142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54823 (local address 127.0.98.62:35717)
0924 04:14:19.697287 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:19.697291 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:19.697302 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:14:19.697378 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:19.697381 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:19.697417 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:19.697479 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:19.697811 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:19.698285 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:19.699026 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:19.699368 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:19.699453 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:19.701515 (+ 2062us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54823: BlockingRecv error: recv got EOF from 127.0.98.3:54823 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
W20260924 04:14:19.701653 9552 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 24 04:14:20 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:14:20 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:14:20.337692 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:20.332531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35717 (local address 127.0.98.1:34121)
0924 04:14:20.332799 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:20.332811 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:20.332905 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:20.333131 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:20.333133 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:20.333298 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:20.333395 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:20.333400 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:20.334149 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:20.334152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:20.334705 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:20.334711 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:20.335167 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:20.335169 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:20.335222 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:20.335740 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:20.335751 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:20.337528 (+ 1777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35717: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":89,"threads_started":1}
W20260924 04:14:20.337850 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:20.332650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34121 (local address 127.0.98.62:35717)
0924 04:14:20.332890 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:20.332893 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:20.332909 (+ 16us) server_negotiation.cc:407] Connection header received
0924 04:14:20.332973 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:20.332975 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:20.333018 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:20.333100 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:20.333504 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:20.334048 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:20.334814 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:20.335054 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:20.335158 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:20.337722 (+ 2564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34121: BlockingRecv error: recv got EOF from 127.0.98.1:34121 (error 108)
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":65,"threads_started":1}
Sep 24 04:14:20 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:14:20 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:14:20.525273 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:20.520031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35717 (local address 127.0.98.2:47625)
0924 04:14:20.520348 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:20.520365 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:20.520488 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:20.520756 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:20.520759 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:20.520961 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:20.521053 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:20.521059 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:20.521787 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:20.521790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:20.522359 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:20.522367 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:20.522779 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:20.522783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:20.522862 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:20.523354 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:20.523365 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:20.525111 (+ 1746us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35717: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":97,"threads_started":1}
W20260924 04:14:20.525367 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:20.520175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47625 (local address 127.0.98.62:35717)
0924 04:14:20.520350 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:20.520354 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:20.520443 (+ 89us) server_negotiation.cc:407] Connection header received
0924 04:14:20.520582 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:20.520585 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:20.520638 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:20.520712 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:20.521177 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:20.521685 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:20.522467 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:20.522680 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:20.522764 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:20.525262 (+ 2498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:47625: BlockingRecv error: recv got EOF from 127.0.98.2:47625 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 24 04:14:20 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:14:20 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:14:20.707252 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:20.702239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35717 (local address 127.0.98.3:58391)
0924 04:14:20.702485 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:20.702498 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:20.702596 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:20.702812 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:20.702832 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:20.702998 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:20.703118 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:20.703123 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:20.703905 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:20.703907 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:20.704470 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:20.704478 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:20.704862 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:20.704864 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:20.704930 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:20.705376 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:20.705387 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:20.707097 (+ 1710us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35717: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":76,"threads_started":1}
W20260924 04:14:20.707398 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:20.702286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58391 (local address 127.0.98.62:35717)
0924 04:14:20.702414 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:20.702419 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:20.702564 (+ 145us) server_negotiation.cc:407] Connection header received
0924 04:14:20.702670 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:20.702672 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:20.702716 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:20.702791 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:20.703207 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:20.703806 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:20.704555 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:20.704776 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:20.704851 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:20.707285 (+ 2434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58391: BlockingRecv error: recv got EOF from 127.0.98.3:58391 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 24 04:14:21 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:14:21 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:14:21.343181 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:21.338463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35717 (local address 127.0.98.1:44051)
0924 04:14:21.338724 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:21.338737 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:21.338836 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:21.339048 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:21.339050 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:21.339210 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:21.339347 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:21.339353 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:21.340057 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:21.340059 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:21.340604 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:21.340611 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:21.340966 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:21.340968 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:21.341022 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:21.341462 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:21.341473 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:21.343044 (+ 1571us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35717: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":87,"threads_started":1}
W20260924 04:14:21.343349 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:21.338581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44051 (local address 127.0.98.62:35717)
0924 04:14:21.338858 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:21.338862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:21.338874 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:14:21.338908 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:21.338910 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:21.338949 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:21.339025 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:21.339447 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:21.339977 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:21.340686 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:21.340893 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:21.340959 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:21.343185 (+ 2226us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44051: BlockingRecv error: recv got EOF from 127.0.98.1:44051 (error 108)
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":139,"threads_started":1}
Sep 24 04:14:21 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:14:21 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:14:21.531651 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:21.526056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35717 (local address 127.0.98.2:46823)
0924 04:14:21.526342 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:21.526355 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:21.526450 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:21.526716 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:21.526719 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:21.526911 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:21.527016 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:21.527022 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:21.527825 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:21.527828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:21.528415 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:21.528425 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:21.528867 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:21.528870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:21.528938 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:21.529482 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:21.529496 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:21.531427 (+ 1931us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35717: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":89,"threads_started":1}
W20260924 04:14:21.531662 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:21.526135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46823 (local address 127.0.98.62:35717)
0924 04:14:21.526271 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:21.526274 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:21.526417 (+ 143us) server_negotiation.cc:407] Connection header received
0924 04:14:21.526520 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:21.526522 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:21.526562 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:21.526629 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:21.527128 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:21.527695 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:21.528534 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:21.528762 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:21.528840 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:21.531551 (+ 2711us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46823: BlockingRecv error: recv got EOF from 127.0.98.2:46823 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 24 04:14:21 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:14:21 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:14:21.713400 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:21.707953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35717 (local address 127.0.98.3:46779)
0924 04:14:21.708224 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:21.708238 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:21.708335 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:21.708537 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:21.708539 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:21.708714 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:21.708806 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:21.708812 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:21.709531 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:21.709534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:21.710153 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:21.710164 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:21.710611 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:21.710614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:21.710684 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:21.711220 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:21.711255 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:21.713221 (+ 1966us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35717: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":79,"threads_started":1}
W20260924 04:14:21.713495 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:21.708087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46779 (local address 127.0.98.62:35717)
0924 04:14:21.708231 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:21.708234 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:21.708286 (+ 52us) server_negotiation.cc:407] Connection header received
0924 04:14:21.708405 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:21.708407 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:21.708450 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:21.708512 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:21.708892 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:21.709414 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:21.710273 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:21.710491 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:21.710568 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:21.713396 (+ 2828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46779: BlockingRecv error: recv got EOF from 127.0.98.3:46779 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 24 04:14:22 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:14:22 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:14:22.349886 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:22.344089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35717 (local address 127.0.98.1:37911)
0924 04:14:22.344361 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:22.344374 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:22.344472 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:22.344805 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:22.344808 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:22.345047 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:22.345198 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:22.345209 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:22.346041 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:22.346045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:22.346804 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:22.346816 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:22.347335 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:22.347339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:22.347428 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:22.347970 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:22.347981 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:22.349750 (+ 1769us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35717: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":84,"threads_started":1}
W20260924 04:14:22.350010 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:22.344205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37911 (local address 127.0.98.62:35717)
0924 04:14:22.344429 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:22.344433 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:22.344446 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:14:22.344592 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:22.344595 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:22.344644 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:22.344738 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:22.345363 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:22.345927 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:22.346932 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:22.347186 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:22.347310 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:22.349880 (+ 2570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37911: BlockingRecv error: recv got EOF from 127.0.98.1:37911 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":55,"threads_started":1}
Sep 24 04:14:22 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:14:22 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:14:22.537469 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:22.532490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35717 (local address 127.0.98.2:44071)
0924 04:14:22.532729 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:22.532744 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:22.532848 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:22.533086 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:22.533088 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:22.533275 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:22.533375 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:22.533382 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:22.534097 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:22.534100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:22.534651 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:22.534660 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:22.535077 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:22.535081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:22.535164 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:22.535652 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:22.535663 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:22.537291 (+ 1628us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35717: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":67,"threads_started":1}
W20260924 04:14:22.537514 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:22.532624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44071 (local address 127.0.98.62:35717)
0924 04:14:22.532759 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:22.532764 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:22.532797 (+ 33us) server_negotiation.cc:407] Connection header received
0924 04:14:22.532941 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:22.532943 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:22.532988 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:22.533052 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:22.533484 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:22.533996 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:22.534759 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:22.534979 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:22.535062 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:22.537421 (+ 2359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44071: BlockingRecv error: recv got EOF from 127.0.98.2:44071 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 24 04:14:22 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:14:22 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:14:22.719007 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:22.714097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35717 (local address 127.0.98.3:52913)
0924 04:14:22.714358 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:22.714371 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:22.714470 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:22.714700 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:22.714702 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:22.714873 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:22.714970 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:22.714976 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:22.715690 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:22.715693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:22.716255 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:22.716262 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:22.716617 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:22.716619 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:22.716678 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:22.717133 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:22.717144 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:22.718888 (+ 1744us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35717: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":91,"threads_started":1}
W20260924 04:14:22.719118 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:22.714171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52913 (local address 127.0.98.62:35717)
0924 04:14:22.714292 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:22.714295 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:22.714433 (+ 138us) server_negotiation.cc:407] Connection header received
0924 04:14:22.714540 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:22.714542 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:22.714586 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:22.714653 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:22.715053 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:22.715594 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:22.716345 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:22.716544 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:22.716610 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:22.719017 (+ 2407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52913: BlockingRecv error: recv got EOF from 127.0.98.3:52913 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 24 04:14:23 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:14:23 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:14:23.361496 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:23.350690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35717 (local address 127.0.98.1:49955)
0924 04:14:23.350952 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:23.350965 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:23.351066 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:23.351380 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:23.351383 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:23.351610 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:23.351712 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:23.351719 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:23.352474 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:23.352477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:23.353056 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:23.353067 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:23.353524 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:23.353527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:23.353603 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:23.359463 (+ 5860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:23.359479 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:23.361370 (+ 1891us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35717: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":74,"threads_started":1}
W20260924 04:14:23.361687 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:23.350698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49955 (local address 127.0.98.62:35717)
0924 04:14:23.350954 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:23.350958 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:23.351032 (+ 74us) server_negotiation.cc:407] Connection header received
0924 04:14:23.351157 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:23.351159 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:23.351212 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:23.351328 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:23.351833 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:23.352349 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:23.353185 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:23.353401 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:23.353474 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:23.361536 (+ 8062us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49955: BlockingRecv error: recv got EOF from 127.0.98.1:49955 (error 108)
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":76,"threads_started":1}
Sep 24 04:14:23 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:14:23 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:14:23.543310 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:23.538205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35717 (local address 127.0.98.2:51151)
0924 04:14:23.538478 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:23.538491 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:23.538598 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:23.538831 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:23.538833 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:23.539026 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:23.539118 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:23.539124 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:23.539907 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:23.539910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:23.540462 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:23.540470 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:23.540896 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:23.540900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:23.540978 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:23.541441 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:23.541452 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:23.543134 (+ 1682us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35717: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":96,"threads_started":1}
W20260924 04:14:23.543413 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:23.538268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51151 (local address 127.0.98.62:35717)
0924 04:14:23.538395 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:23.538400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:23.538558 (+ 158us) server_negotiation.cc:407] Connection header received
0924 04:14:23.538686 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:23.538689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:23.538734 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:23.538799 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:23.539266 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:23.539800 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:23.540571 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:23.540796 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:23.540881 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:23.543316 (+ 2435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51151: BlockingRecv error: recv got EOF from 127.0.98.2:51151 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 24 04:14:23 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:14:23 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:14:23.724908 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:23.719733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35717 (local address 127.0.98.3:50351)
0924 04:14:23.719989 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:23.720001 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:23.720093 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:23.720304 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:23.720306 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:23.720482 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:23.720583 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:23.720589 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:23.721393 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:23.721396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:23.721951 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:23.721960 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:23.722429 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:23.722433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:23.722519 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:23.723019 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:23.723032 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:23.724798 (+ 1766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35717: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":87,"threads_started":1}
W20260924 04:14:23.725035 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:23.719832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50351 (local address 127.0.98.62:35717)
0924 04:14:23.719953 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:23.719957 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:23.720060 (+ 103us) server_negotiation.cc:407] Connection header received
0924 04:14:23.720168 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:23.720170 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:23.720214 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:23.720283 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:23.720720 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:23.721282 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:23.722068 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:23.722319 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:23.722412 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:23.724921 (+ 2509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50351: BlockingRecv error: recv got EOF from 127.0.98.3:50351 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 24 04:14:24 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:14:24 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:14:24.368039 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:24.362357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35717 (local address 127.0.98.1:56817)
0924 04:14:24.362631 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:24.362645 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:24.362732 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:24.363018 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:24.363021 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:24.363192 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:24.363327 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:24.363332 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:24.364046 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:24.364049 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:24.364754 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:24.364765 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:24.365164 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:24.365166 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:24.365223 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:24.365823 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:24.365835 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:24.367926 (+ 2091us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35717: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":71,"threads_started":1}
W20260924 04:14:24.368168 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:24.362451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56817 (local address 127.0.98.62:35717)
0924 04:14:24.362676 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:24.362680 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:24.362694 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:14:24.362811 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:24.362814 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:24.362858 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:24.362939 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:24.363423 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:24.363944 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:24.364857 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:24.365077 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:24.365152 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:24.368040 (+ 2888us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56817: BlockingRecv error: recv got EOF from 127.0.98.1:56817 (error 108)
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":66,"threads_started":1}
Sep 24 04:14:24 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:14:24 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:14:24.549551 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:24.544165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35717 (local address 127.0.98.2:42047)
0924 04:14:24.544417 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:24.544430 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:24.544521 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:24.544786 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:24.544788 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:24.544981 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:24.545085 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:24.545092 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:24.545823 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:24.545826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:24.546438 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:24.546448 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:24.546986 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:24.546989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:24.547058 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:24.547575 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:24.547587 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:24.549437 (+ 1850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35717: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":81,"threads_started":1}
W20260924 04:14:24.549674 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:24.544278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42047 (local address 127.0.98.62:35717)
0924 04:14:24.544412 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:24.544415 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:24.544488 (+ 73us) server_negotiation.cc:407] Connection header received
0924 04:14:24.544589 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:24.544591 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:24.544631 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:24.544696 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:24.545202 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:24.545713 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:24.546565 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:24.546877 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:24.546964 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:24.549556 (+ 2592us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42047: BlockingRecv error: recv got EOF from 127.0.98.2:42047 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 24 04:14:24 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:14:24 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:14:24.731731 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:24.725603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35717 (local address 127.0.98.3:58873)
0924 04:14:24.725875 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:24.725890 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:24.726000 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:24.726283 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:24.726286 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:24.726490 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:24.726606 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:24.726613 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:24.727667 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:24.727670 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:24.728551 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:24.728563 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:24.729114 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:24.729117 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:24.729196 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:24.729747 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:24.729759 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:24.731571 (+ 1812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35717: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":103,"threads_started":1}
W20260924 04:14:24.731874 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:24.725673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58873 (local address 127.0.98.62:35717)
0924 04:14:24.725795 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:24.725799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:24.725954 (+ 155us) server_negotiation.cc:407] Connection header received
0924 04:14:24.726109 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:24.726113 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:24.726175 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:24.726247 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:24.726735 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:24.727551 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:24.728667 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:24.729003 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:24.729097 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:24.731774 (+ 2677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58873: BlockingRecv error: recv got EOF from 127.0.98.3:58873 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 24 04:14:25 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:14:25 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:14:25.373574 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:25.368662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35717 (local address 127.0.98.1:52481)
0924 04:14:25.368895 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:25.368909 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:25.368994 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:25.369209 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:25.369211 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:25.369381 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:25.369472 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:25.369477 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:25.370155 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:25.370158 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:25.370762 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:25.370771 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:25.371202 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:25.371204 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:25.371302 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:25.371783 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:25.371794 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:25.373462 (+ 1668us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35717: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":66,"threads_started":1}
W20260924 04:14:25.373701 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:25.368719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52481 (local address 127.0.98.62:35717)
0924 04:14:25.368954 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:25.368957 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:25.368970 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:14:25.369062 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:25.369064 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:25.369104 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:25.369187 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:25.369551 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:25.370073 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:25.370877 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:25.371098 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:25.371191 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:25.373586 (+ 2395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52481: BlockingRecv error: recv got EOF from 127.0.98.1:52481 (error 108)
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":95,"threads_started":1}
Sep 24 04:14:25 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:14:25 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:14:25.555727 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:25.550306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35717 (local address 127.0.98.2:46321)
0924 04:14:25.550565 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:25.550578 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:25.550690 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:25.550918 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:25.550920 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:25.551095 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:25.551208 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:25.551214 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:25.552159 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:25.552162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:25.552849 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:25.552860 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:25.553305 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:25.553309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:25.553379 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:25.553841 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:25.553853 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:25.555584 (+ 1731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35717: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":84,"threads_started":1}
W20260924 04:14:25.555819 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:25.550403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46321 (local address 127.0.98.62:35717)
0924 04:14:25.550526 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:25.550530 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:25.550643 (+ 113us) server_negotiation.cc:407] Connection header received
0924 04:14:25.550762 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:25.550764 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:25.550805 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:25.550877 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:25.551388 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:25.552035 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:25.552952 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:25.553177 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:25.553256 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:25.555723 (+ 2467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46321: BlockingRecv error: recv got EOF from 127.0.98.2:46321 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 24 04:14:25 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:14:25 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:14:25.738039 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:25.732541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35717 (local address 127.0.98.3:57273)
0924 04:14:25.732843 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:25.732861 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:25.732990 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:25.733260 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:25.733263 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:25.733483 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:25.733585 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:25.733592 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:25.734360 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:25.734364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:25.735043 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:25.735053 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:25.735584 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:25.735587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:25.735677 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:25.736255 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:25.736267 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:25.737915 (+ 1648us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35717: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":95,"threads_started":1}
W20260924 04:14:25.738152 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:25.732622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57273 (local address 127.0.98.62:35717)
0924 04:14:25.732800 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:25.732805 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:25.732941 (+ 136us) server_negotiation.cc:407] Connection header received
0924 04:14:25.733089 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:25.733093 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:25.733146 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:25.733227 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:25.733711 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:25.734245 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:25.735172 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:25.735457 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:25.735573 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:25.738051 (+ 2478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:57273: BlockingRecv error: recv got EOF from 127.0.98.3:57273 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 24 04:14:26 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:14:26 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:14:26.379383 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:26.374258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35717 (local address 127.0.98.1:35383)
0924 04:14:26.374535 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:26.374548 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:26.374644 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:26.374860 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:26.374862 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:26.375023 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:26.375115 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:26.375132 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:26.375846 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:26.375848 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:26.376447 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:26.376459 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:26.377023 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:26.377027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:26.377111 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:26.377600 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:26.377611 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:26.379269 (+ 1658us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35717: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":67,"threads_started":1}
W20260924 04:14:26.379518 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:26.374326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35383 (local address 127.0.98.62:35717)
0924 04:14:26.374558 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:26.374561 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:26.374595 (+ 34us) server_negotiation.cc:407] Connection header received
0924 04:14:26.374719 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:26.374721 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:26.374763 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:26.374838 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:26.375210 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:26.375754 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:26.376597 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:26.376915 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:26.377011 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:26.379393 (+ 2382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35383: BlockingRecv error: recv got EOF from 127.0.98.1:35383 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":87,"threads_started":1}
Sep 24 04:14:26 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:14:26 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:14:26.561635 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:26.556463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35717 (local address 127.0.98.2:45777)
0924 04:14:26.556761 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:26.556774 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:26.556872 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:26.557089 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:26.557091 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:26.557277 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:26.557371 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:26.557377 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:26.558068 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:26.558071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:26.558635 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:26.558643 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:26.559035 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:26.559038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:26.559100 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:26.559635 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:26.559647 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:26.561484 (+ 1837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35717: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":95,"threads_started":1}
W20260924 04:14:26.561730 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:26.556529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45777 (local address 127.0.98.62:35717)
0924 04:14:26.556703 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:26.556707 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:26.556840 (+ 133us) server_negotiation.cc:407] Connection header received
0924 04:14:26.556949 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:26.556951 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:26.556996 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:26.557065 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:26.557458 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:26.557986 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:26.558735 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:26.558946 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:26.559019 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:26.561628 (+ 2609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45777: BlockingRecv error: recv got EOF from 127.0.98.2:45777 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 24 04:14:26 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:14:26 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:14:26.743625 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:26.738715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35717 (local address 127.0.98.3:50887)
0924 04:14:26.738979 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:26.738992 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:26.739087 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:26.739329 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:26.739331 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:26.739499 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:26.739593 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:26.739599 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:26.740271 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:26.740273 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:26.740819 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:26.740826 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:26.741202 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:26.741204 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:26.741286 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:26.741743 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:26.741754 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:26.743508 (+ 1754us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35717: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":73,"threads_started":1}
W20260924 04:14:26.743728 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:26.738786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50887 (local address 127.0.98.62:35717)
0924 04:14:26.738923 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:26.738926 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:26.739079 (+ 153us) server_negotiation.cc:407] Connection header received
0924 04:14:26.739152 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:26.739155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:26.739204 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:26.739303 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:26.739687 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:26.740194 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:26.740895 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:26.741094 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:26.741190 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:26.743632 (+ 2442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50887: BlockingRecv error: recv got EOF from 127.0.98.3:50887 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 24 04:14:27 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:14:27 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:14:27.385090 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:27.380080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35717 (local address 127.0.98.1:47161)
0924 04:14:27.380361 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:27.380378 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:27.380473 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:27.380679 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:27.380681 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:27.380851 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:27.380945 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:27.380950 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:27.381643 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:27.381646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:27.382213 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:27.382222 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:27.382690 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:27.382694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:27.382771 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:27.383260 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:27.383273 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:27.384976 (+ 1703us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35717: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":80,"threads_started":1}
W20260924 04:14:27.385212 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:27.380225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47161 (local address 127.0.98.62:35717)
0924 04:14:27.380458 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:27.380461 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:27.380473 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:14:27.380548 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:27.380550 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:27.380589 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:27.380652 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:27.381027 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:27.381530 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:27.382325 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:27.382550 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:27.382623 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:27.385101 (+ 2478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47161: BlockingRecv error: recv got EOF from 127.0.98.1:47161 (error 108)
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":72,"threads_started":1}
Sep 24 04:14:27 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:14:27 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:14:27.567577 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:27.562415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35717 (local address 127.0.98.2:42897)
0924 04:14:27.562678 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:27.562692 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:27.562790 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:27.562980 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:27.562982 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:27.563149 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:27.563268 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:27.563273 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:27.564142 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:27.564146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:27.564773 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:27.564782 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:27.565169 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:27.565171 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:27.565226 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:27.565718 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:27.565729 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:27.567464 (+ 1735us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35717: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":66,"threads_started":1}
W20260924 04:14:27.567709 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:27.562495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42897 (local address 127.0.98.62:35717)
0924 04:14:27.562653 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:27.562663 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:27.562756 (+ 93us) server_negotiation.cc:407] Connection header received
0924 04:14:27.562867 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:27.562870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:27.562911 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:27.562964 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:27.563347 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:27.564036 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:27.564872 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:27.565090 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:27.565162 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:27.567599 (+ 2437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42897: BlockingRecv error: recv got EOF from 127.0.98.2:42897 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 24 04:14:27 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:14:27 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:14:27.750159 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:27.744490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35717 (local address 127.0.98.3:57987)
0924 04:14:27.744788 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:27.744802 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:27.744924 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:27.745165 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:27.745168 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:27.745365 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:27.745453 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:27.745459 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:27.746199 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:27.746203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:27.746765 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:27.746772 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:27.747325 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:27.747329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:27.747430 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:27.748073 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:27.748088 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:27.750010 (+ 1922us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35717: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":126,"threads_started":1}
W20260924 04:14:27.750258 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:27.744620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57987 (local address 127.0.98.62:35717)
0924 04:14:27.744781 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:27.744786 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:27.744878 (+ 92us) server_negotiation.cc:407] Connection header received
0924 04:14:27.745012 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:27.745015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:27.745063 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:27.745139 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:27.745566 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:27.746089 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:27.746872 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:27.747161 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:27.747372 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:27.750148 (+ 2776us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:57987: BlockingRecv error: recv got EOF from 127.0.98.3:57987 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 24 04:14:28 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:14:28 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:14:28.396905 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:28.385894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35717 (local address 127.0.98.1:38193)
0924 04:14:28.386176 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:28.386191 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:28.386283 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:28.386508 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:28.386510 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:28.386683 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:28.386776 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:28.386781 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:28.387526 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:28.387529 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:28.388100 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:28.388108 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:28.388618 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:28.388621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:28.388689 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:28.394857 (+ 6168us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:28.394872 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:28.396782 (+ 1910us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35717: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":84,"threads_started":1}
W20260924 04:14:28.397113 9590 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:28.386020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38193 (local address 127.0.98.62:35717)
0924 04:14:28.386292 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:28.386296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:28.386331 (+ 35us) server_negotiation.cc:407] Connection header received
0924 04:14:28.386369 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:28.386372 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:28.386414 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:28.386480 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:28.386863 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:28.387428 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:28.388261 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:28.388496 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:28.388580 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:28.396971 (+ 8391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38193: BlockingRecv error: recv got EOF from 127.0.98.1:38193 (error 108)
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":76,"threads_started":1}
Sep 24 04:14:28 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:14:28 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:14:28.573486 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:28.568342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35717 (local address 127.0.98.2:49977)
0924 04:14:28.568618 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:28.568630 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:28.568734 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:28.568974 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:28.568977 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:28.569183 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:28.569295 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:28.569302 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:28.570050 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:28.570054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:28.570621 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:28.570630 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:28.571055 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:28.571059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:28.571133 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:28.571641 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:28.571652 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:28.573372 (+ 1720us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35717: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":76,"threads_started":1}
W20260924 04:14:28.573740 9590 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:28.568399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49977 (local address 127.0.98.62:35717)
0924 04:14:28.568525 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:28.568530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:28.568694 (+ 164us) server_negotiation.cc:407] Connection header received
0924 04:14:28.568830 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:28.568833 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:28.568880 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:28.568942 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:28.569413 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:28.569953 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:28.570727 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:28.570957 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:28.571039 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:28.573570 (+ 2531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49977: BlockingRecv error: recv got EOF from 127.0.98.2:49977 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 24 04:14:28 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:14:28 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:14:28.756510 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:28.750968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35717 (local address 127.0.98.3:52303)
0924 04:14:28.751221 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:28.751259 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:28.751382 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:28.751640 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:28.751642 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:28.751810 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:28.751900 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:28.751905 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:28.752739 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:28.752742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:28.753287 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:28.753294 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:28.753849 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:28.753853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:28.753947 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:28.754498 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:28.754512 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:28.756364 (+ 1852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35717: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":89,"threads_started":1}
W20260924 04:14:28.756595 9590 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:28.751057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52303 (local address 127.0.98.62:35717)
0924 04:14:28.751205 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:28.751209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:28.751325 (+ 116us) server_negotiation.cc:407] Connection header received
0924 04:14:28.751491 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:28.751494 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:28.751543 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:28.751620 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:28.752028 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:28.752628 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:28.753414 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:28.753704 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:28.753804 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:28.756498 (+ 2694us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52303: BlockingRecv error: recv got EOF from 127.0.98.3:52303 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 24 04:14:29 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:14:29 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:14:29.402935 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:29.397681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35717 (local address 127.0.98.1:47667)
0924 04:14:29.398043 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:29.398058 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:29.398152 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:29.398436 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:29.398439 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:29.398642 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:29.398746 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:29.398753 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:29.399502 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:29.399505 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:29.400077 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:29.400085 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:29.400503 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:29.400505 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:29.400572 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:29.401079 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:29.401091 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:29.402765 (+ 1674us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35717: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":81,"threads_started":1}
W20260924 04:14:29.403019 9594 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:29.397721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47667 (local address 127.0.98.62:35717)
0924 04:14:29.398010 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:29.398014 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:29.398118 (+ 104us) server_negotiation.cc:407] Connection header received
0924 04:14:29.398223 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:29.398225 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:29.398267 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:29.398352 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:29.398843 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:29.399388 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:29.400187 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:29.400405 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:29.400471 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:29.402911 (+ 2440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47667: BlockingRecv error: recv got EOF from 127.0.98.1:47667 (error 108)
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":86,"threads_started":1}
Sep 24 04:14:29 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:14:29 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:14:29.579453 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:29.574292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35717 (local address 127.0.98.2:44171)
0924 04:14:29.574543 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:29.574556 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:29.574650 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:29.574856 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:29.574858 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:29.575028 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:29.575133 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:29.575138 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:29.575846 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:29.575849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:29.576427 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:29.576435 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:29.576833 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:29.576837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:29.576905 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:29.577477 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:29.577488 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:29.579335 (+ 1847us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35717: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":68,"threads_started":1}
W20260924 04:14:29.579596 9594 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:29.574357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44171 (local address 127.0.98.62:35717)
0924 04:14:29.574491 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:29.574496 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:29.574616 (+ 120us) server_negotiation.cc:407] Connection header received
0924 04:14:29.574722 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:29.574724 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:29.574764 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:29.574834 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:29.575214 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:29.575750 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:29.576515 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:29.576723 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:29.576794 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:29.579487 (+ 2693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44171: BlockingRecv error: recv got EOF from 127.0.98.2:44171 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 24 04:14:29 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:14:29 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:14:29.762928 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:29.757383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35717 (local address 127.0.98.3:56297)
0924 04:14:29.757632 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:29.757645 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:29.757760 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:29.757971 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:29.757973 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:29.758142 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:29.758236 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:29.758242 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:29.759029 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:29.759032 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:29.759764 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:29.759777 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:29.760238 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:29.760242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:29.760325 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:29.760884 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:29.760898 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:29.762814 (+ 1916us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35717: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":68,"threads_started":1}
W20260924 04:14:29.763036 9594 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:29.757528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56297 (local address 127.0.98.62:35717)
0924 04:14:29.757642 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:29.757645 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:29.757694 (+ 49us) server_negotiation.cc:407] Connection header received
0924 04:14:29.757842 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:29.757844 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:29.757886 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:29.757943 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:29.758331 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:29.758877 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:29.759891 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:29.760110 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:29.760225 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:29.762942 (+ 2717us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56297: BlockingRecv error: recv got EOF from 127.0.98.3:56297 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 24 04:14:30 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:14:30 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:14:30.409309 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:30.403774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35717 (local address 127.0.98.1:42083)
0924 04:14:30.404039 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:30.404055 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:30.404144 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:30.404381 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:30.404384 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:30.404548 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:30.404639 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:30.404643 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:30.405372 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:30.405374 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:30.405969 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:30.405977 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:30.406438 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:30.406440 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:30.406503 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:30.407025 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:30.407040 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:30.409168 (+ 2128us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35717: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":70,"threads_started":1}
W20260924 04:14:30.409416 9598 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:30.403880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42083 (local address 127.0.98.62:35717)
0924 04:14:30.404128 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:30.404132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:30.404147 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:14:30.404230 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:30.404232 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:30.404276 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:30.404358 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:30.404752 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:30.405289 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:30.406070 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:30.406336 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:30.406424 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:30.409288 (+ 2864us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42083: BlockingRecv error: recv got EOF from 127.0.98.1:42083 (error 108)
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":69,"threads_started":1}
Sep 24 04:14:30 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:14:30 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:14:30.585414 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:30.580232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35717 (local address 127.0.98.2:50069)
0924 04:14:30.580557 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:30.580573 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:30.580739 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:30.581016 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:30.581019 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:30.581205 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:30.581314 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:30.581322 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:30.582130 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:30.582132 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:30.582683 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:30.582691 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:30.583120 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:30.583122 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:30.583178 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:30.583657 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:30.583668 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:30.585280 (+ 1612us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35717: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":75,"threads_started":1}
W20260924 04:14:30.585534 9598 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:30.580366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50069 (local address 127.0.98.62:35717)
0924 04:14:30.580504 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:30.580508 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:30.580651 (+ 143us) server_negotiation.cc:407] Connection header received
0924 04:14:30.580841 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:30.580844 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:30.580901 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:30.580976 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:30.581435 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:30.582035 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:30.582803 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:30.583028 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:30.583121 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:30.585418 (+ 2297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50069: BlockingRecv error: recv got EOF from 127.0.98.2:50069 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 24 04:14:30 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:14:30 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:14:30.769327 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:30.763650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35717 (local address 127.0.98.3:58707)
0924 04:14:30.763883 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:30.763896 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:30.763993 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:30.764206 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:30.764209 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:30.764381 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:30.764473 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:30.764478 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:30.765200 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:30.765203 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:30.765792 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:30.765800 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:30.766334 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:30.766337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:30.766415 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:30.766965 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:30.766981 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:30.769146 (+ 2165us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35717: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":71,"threads_started":1}
W20260924 04:14:30.769577 9598 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:30.763719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58707 (local address 127.0.98.62:35717)
0924 04:14:30.763878 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:30.763883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:30.763958 (+ 75us) server_negotiation.cc:407] Connection header received
0924 04:14:30.764077 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:30.764079 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:30.764125 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:30.764184 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:30.764564 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:30.765118 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:30.765901 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:30.766217 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:30.766320 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:30.769303 (+ 2983us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58707: BlockingRecv error: recv got EOF from 127.0.98.3:58707 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 24 04:14:31 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:14:31 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:14:31.415207 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:31.410032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35717 (local address 127.0.98.1:45881)
0924 04:14:31.410333 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:31.410347 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:31.410456 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:31.410702 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:31.410704 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:31.410886 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:31.410988 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:31.410994 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:31.411745 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:31.411749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:31.412332 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:31.412342 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:31.412810 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:31.412813 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:31.412883 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:31.413360 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:31.413372 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:31.415084 (+ 1712us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35717: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":82,"threads_started":1}
W20260924 04:14:31.415385 9602 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:31.410171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45881 (local address 127.0.98.62:35717)
0924 04:14:31.410402 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:31.410406 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:31.410420 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:14:31.410540 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:31.410542 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:31.410582 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:31.410657 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:31.411079 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:31.411633 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:31.412452 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:31.412678 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:31.412774 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:31.415219 (+ 2445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45881: BlockingRecv error: recv got EOF from 127.0.98.1:45881 (error 108)
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":95,"threads_started":1}
Sep 24 04:14:31 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:14:31 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:14:31.591507 9603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:31.586124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35717 (local address 127.0.98.2:46729)
0924 04:14:31.586404 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:31.586419 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:31.586529 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:31.586766 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:31.586769 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:31.586981 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:31.587076 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:31.587082 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:31.587867 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:31.587871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:31.588413 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:31.588421 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:31.588836 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:31.588840 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:31.588919 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:31.589423 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:31.589436 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:31.591384 (+ 1948us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35717: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":105,"threads_started":1}
W20260924 04:14:31.591629 9602 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:31.586243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46729 (local address 127.0.98.62:35717)
0924 04:14:31.586390 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:31.586394 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:31.586487 (+ 93us) server_negotiation.cc:407] Connection header received
0924 04:14:31.586615 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:31.586618 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:31.586663 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:31.586729 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:31.587192 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:31.587755 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:31.588521 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:31.588733 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:31.588816 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:31.591525 (+ 2709us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46729: BlockingRecv error: recv got EOF from 127.0.98.2:46729 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 24 04:14:31 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:14:31 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:14:31.775786 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:31.770118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35717 (local address 127.0.98.3:60041)
0924 04:14:31.770519 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:31.770533 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:31.770633 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:31.770974 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:31.770977 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:31.771156 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:31.771332 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:31.771352 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:31.772171 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:31.772174 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:31.772725 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:31.772733 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:31.773210 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:31.773212 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:31.773274 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:31.773786 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:31.773796 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:31.775665 (+ 1869us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35717: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":67,"threads_started":1}
W20260924 04:14:31.775985 9602 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:31.770248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60041 (local address 127.0.98.62:35717)
0924 04:14:31.770385 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:31.770388 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:31.770607 (+ 219us) server_negotiation.cc:407] Connection header received
0924 04:14:31.770774 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:31.770778 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:31.770842 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:31.770935 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:31.771449 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:31.772073 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:31.772851 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:31.773079 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:31.773194 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:31.775864 (+ 2670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60041: BlockingRecv error: recv got EOF from 127.0.98.3:60041 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 24 04:14:32 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:14:32 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:14:32.421900 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:32.416063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35717 (local address 127.0.98.1:48463)
0924 04:14:32.416513 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:32.416529 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:32.416670 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:32.417066 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:32.417068 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:32.417252 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:32.417386 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:32.417395 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:32.418284 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:32.418288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:32.418849 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:32.418858 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:32.419270 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:32.419274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:32.419341 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:32.419797 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:32.419826 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:32.421774 (+ 1948us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35717: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":74,"threads_started":1}
W20260924 04:14:32.422019 9606 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:32.416222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48463 (local address 127.0.98.62:35717)
0924 04:14:32.416467 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:32.416473 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:32.416670 (+ 197us) server_negotiation.cc:407] Connection header received
0924 04:14:32.416765 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:32.416767 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:32.416821 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:32.416929 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:32.417521 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:32.418181 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:32.418945 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:32.419160 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:32.419266 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:32.421909 (+ 2643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48463: BlockingRecv error: recv got EOF from 127.0.98.1:48463 (error 108)
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":83,"threads_started":1}
Sep 24 04:14:32 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:14:32 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:14:32.597422 9607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:32.592264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35717 (local address 127.0.98.2:44767)
0924 04:14:32.592525 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:32.592540 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:32.592639 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:32.592844 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:32.592846 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:32.593015 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:32.593124 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:32.593129 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:32.593860 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:32.593864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:32.594432 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:32.594441 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:32.594862 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:32.594864 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:32.594934 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:32.595446 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:32.595459 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:32.597268 (+ 1809us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35717: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":68,"threads_started":1}
W20260924 04:14:32.597621 9606 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:32.592430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44767 (local address 127.0.98.62:35717)
0924 04:14:32.592560 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:32.592563 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:32.592590 (+ 27us) server_negotiation.cc:407] Connection header received
0924 04:14:32.592711 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:32.592714 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:32.592755 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:32.592819 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:32.593209 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:32.593741 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:32.594548 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:32.594772 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:32.594851 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:32.597413 (+ 2562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44767: BlockingRecv error: recv got EOF from 127.0.98.2:44767 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 24 04:14:32 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:14:32 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:14:32.781664 9608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:32.776585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35717 (local address 127.0.98.3:33641)
0924 04:14:32.776903 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:32.776918 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:32.777015 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:32.777263 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:32.777265 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:32.777485 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:32.777599 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:32.777605 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:32.778311 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:32.778313 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:32.778881 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:32.778888 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:32.779263 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:32.779266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:32.779323 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:32.779818 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:32.779830 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:32.781550 (+ 1720us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35717: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":75,"threads_started":1}
W20260924 04:14:32.781805 9606 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:32.776637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33641 (local address 127.0.98.62:35717)
0924 04:14:32.776740 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:32.776745 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:32.776991 (+ 246us) server_negotiation.cc:407] Connection header received
0924 04:14:32.777096 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:32.777099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:32.777149 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:32.777229 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:32.777698 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:32.778223 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:32.778960 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:32.779165 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:32.779258 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:32.781687 (+ 2429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33641: BlockingRecv error: recv got EOF from 127.0.98.3:33641 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 24 04:14:33 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:14:33 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:14:33.432343 9609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:33.422807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35717 (local address 127.0.98.1:48545)
0924 04:14:33.423090 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:33.423105 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:33.423216 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:33.423536 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:33.423539 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:33.423735 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:33.423855 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:33.423863 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:33.424811 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:33.424814 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:33.425396 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:33.425405 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:33.425791 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:33.425793 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:33.425849 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:33.430198 (+ 4349us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:33.430214 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:33.432182 (+ 1968us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35717: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":93,"threads_started":1}
W20260924 04:14:33.432497 9610 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:33.422886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48545 (local address 127.0.98.62:35717)
0924 04:14:33.423161 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:33.423165 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:33.423180 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:14:33.423352 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:33.423355 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:33.423399 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:33.423483 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:33.423974 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:33.424708 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:33.425496 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:33.425711 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:33.425781 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:33.432363 (+ 6582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48545: BlockingRecv error: recv got EOF from 127.0.98.1:48545 (error 108)
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":67,"threads_started":1}
Sep 24 04:14:33 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:14:33 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:14:33.603729 9611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:33.598246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35717 (local address 127.0.98.2:57151)
0924 04:14:33.598583 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:33.598600 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:33.598729 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:33.599006 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:33.599009 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:33.599201 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:33.599320 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:33.599331 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:33.600112 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:33.600115 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:33.600729 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:33.600740 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:33.601267 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:33.601271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:33.601356 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:33.601800 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:33.601811 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:33.603584 (+ 1773us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35717: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":107,"threads_started":1}
W20260924 04:14:33.603819 9610 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:33.598413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57151 (local address 127.0.98.62:35717)
0924 04:14:33.598585 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:33.598589 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:33.598668 (+ 79us) server_negotiation.cc:407] Connection header received
0924 04:14:33.598850 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:33.598853 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:33.598895 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:33.598964 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:33.599439 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:33.600006 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:33.600870 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:33.601150 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:33.601250 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:33.603715 (+ 2465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57151: BlockingRecv error: recv got EOF from 127.0.98.2:57151 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 24 04:14:33 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:14:33 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:14:33.787433 9612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:33.782344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35717 (local address 127.0.98.3:36075)
0924 04:14:33.782595 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:33.782608 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:33.782692 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:33.782999 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:33.783002 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:33.783173 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:33.783300 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:33.783308 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:33.784025 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:33.784027 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:33.784567 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:33.784574 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:33.785029 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:33.785031 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:33.785088 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:33.785563 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:33.785575 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:33.787298 (+ 1723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35717: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":84,"threads_started":1}
W20260924 04:14:33.787541 9610 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:33.782472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36075 (local address 127.0.98.62:35717)
0924 04:14:33.782626 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:33.782630 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:33.782647 (+ 17us) server_negotiation.cc:407] Connection header received
0924 04:14:33.782820 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:33.782823 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:33.782872 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:33.782944 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:33.783402 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:33.783944 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:33.784715 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:33.784936 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:33.785022 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:33.787439 (+ 2417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36075: BlockingRecv error: recv got EOF from 127.0.98.3:36075 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 24 04:14:34 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:14:34 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:14:34.438779 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:34.433340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35717 (local address 127.0.98.1:52463)
0924 04:14:34.433660 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:34.433675 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:34.433778 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:34.434032 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:34.434035 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:34.434202 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:34.434297 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:34.434304 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:34.435110 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:34.435112 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:34.435741 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:34.435751 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:34.436209 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:34.436211 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:34.436269 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:34.436767 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:34.436779 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:34.438581 (+ 1802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35717: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":73,"threads_started":1}
W20260924 04:14:34.438982 9614 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:34.433481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52463 (local address 127.0.98.62:35717)
0924 04:14:34.433770 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:34.433773 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:34.433788 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:14:34.433847 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:34.433849 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:34.433890 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:34.433970 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:34.434447 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:34.435002 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:34.435872 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:34.436114 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:34.436205 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:34.438729 (+ 2524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52463: BlockingRecv error: recv got EOF from 127.0.98.1:52463 (error 108)
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":75,"threads_started":1}
Sep 24 04:14:34 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:14:34 dist-test-slave-p401 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:14:34.609834 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:34.604575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35717 (local address 127.0.98.2:40273)
0924 04:14:34.604896 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:34.604910 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:34.605014 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:34.605239 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:34.605241 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:34.605437 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:34.605526 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:34.605532 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:34.606321 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:34.606324 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:34.606913 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:34.606921 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:34.607389 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:34.607393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:34.607469 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:34.607942 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:34.607954 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:34.609681 (+ 1727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35717: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":82,"threads_started":1}
W20260924 04:14:34.609896 9614 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:34.604625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40273 (local address 127.0.98.62:35717)
0924 04:14:34.604760 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:34.604764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:34.604966 (+ 202us) server_negotiation.cc:407] Connection header received
0924 04:14:34.605095 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:34.605098 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:34.605146 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:34.605208 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:34.605632 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:34.606196 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:34.607024 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:34.607277 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:34.607369 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:34.609808 (+ 2439us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:40273: BlockingRecv error: recv got EOF from 127.0.98.2:40273 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 04:14:34.696852 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 9139
I20260924 04:14:34.701845 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 9291
I20260924 04:14:34.706314 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 9423
I20260924 04:14:34.710590 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 9069
2026-09-24T04:14:34Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17858 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-24T04:14:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:14:34Z Disabled control of system clock
I20260924 04:14:34.738479 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45563
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45563
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 04:14:34.847141 9619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:14:34.847447 9619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:14:34.847496 9619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:14:34.851544 9619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:14:34.851609 9619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:14:34.851630 9619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:14:34.851652 9619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:14:34.851675 9619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:14:34.856417 9619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45563
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:14:34.857447 9619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:14:34.858774 9619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:14:34.867408 9624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:34.867462 9619 server_base.cc:1034] running on GCE node
W20260924 04:14:34.867417 9627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:34.867419 9625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:34.868176 9619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:14:34.869247 9619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:14:34.870415 9619 hybrid_clock.cc:648] HybridClock initialized: now 1790223274870390 us; error 38 us; skew 500 ppm
I20260924 04:14:34.872278 9619 webserver.cc:460] Webserver started at http://127.0.98.62:46825/ using document root <none> and password file <none>
I20260924 04:14:34.872833 9619 fs_manager.cc:359] Metadata directory not provided
I20260924 04:14:34.872890 9619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:14:34.873086 9619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:14:34.875080 9619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/master-0/data/instance:
uuid: "9594d5f86c194ee8ab605c1e42e4adcf"
format_stamp: "Formatted at 2026-09-24 04:14:34 on dist-test-slave-p401"
server_key: "7d49bfce4253b05d836e4d09012873d6"
server_key_iv: "97db107b1f2cb5aeddcbcace511c4a5a"
server_key_version: "encryptionkey@0"
I20260924 04:14:34.875640 9619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/master-0/wal/instance:
uuid: "9594d5f86c194ee8ab605c1e42e4adcf"
format_stamp: "Formatted at 2026-09-24 04:14:34 on dist-test-slave-p401"
server_key: "7d49bfce4253b05d836e4d09012873d6"
server_key_iv: "97db107b1f2cb5aeddcbcace511c4a5a"
server_key_version: "encryptionkey@0"
I20260924 04:14:34.879029 9619 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:14:34.881335 9633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:34.882478 9619 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:14:34.882582 9619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/master-0/wal
uuid: "9594d5f86c194ee8ab605c1e42e4adcf"
format_stamp: "Formatted at 2026-09-24 04:14:34 on dist-test-slave-p401"
server_key: "7d49bfce4253b05d836e4d09012873d6"
server_key_iv: "97db107b1f2cb5aeddcbcace511c4a5a"
server_key_version: "encryptionkey@0"
I20260924 04:14:34.882673 9619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:14:34.892908 9619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:14:34.893520 9619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:14:34.893677 9619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:14:34.900123 9619 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45563
I20260924 04:14:34.900182 9685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45563 every 8 connection(s)
I20260924 04:14:34.901024 9619 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:14:34.903825 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 9619
I20260924 04:14:34.903918 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/master-0/wal/instance
I20260924 04:14:34.904127 9686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:14:34.904170 392 external_mini_cluster.cc:1428] Setting key 576395e468799a77a94467232b0259fc
I20260924 04:14:34.911123 9686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9594d5f86c194ee8ab605c1e42e4adcf: Bootstrap starting.
I20260924 04:14:34.913738 9686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9594d5f86c194ee8ab605c1e42e4adcf: Neither blocks nor log segments found. Creating new log.
I20260924 04:14:34.914669 9686 log.cc:824] T 00000000000000000000000000000000 P 9594d5f86c194ee8ab605c1e42e4adcf: Log is configured to *not* fsync() on all Append() calls
I20260924 04:14:34.914865 9689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:34.905619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35832 (local address 127.0.98.62:45563)
0924 04:14:34.906290 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:34.906308 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:34.906351 (+ 43us) server_negotiation.cc:407] Connection header received
0924 04:14:34.907403 (+ 1052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:34.907428 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:34.907942 (+ 514us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:34.908322 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:34.908747 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:34.910215 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:34.911176 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:34.911874 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:34.912024 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:34.912871 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:34.912877 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:34.912891 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:14:34.913370 (+ 479us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:34.913438 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:14:34.913463 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:34.913556 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:34.913669 (+ 113us) server_negotiation.cc:299] Negotiation successful
0924 04:14:34.913939 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":107,"threads_started":1}
I20260924 04:14:34.916736 9686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9594d5f86c194ee8ab605c1e42e4adcf: No bootstrap required, opened a new log
I20260924 04:14:34.920166 9686 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9594d5f86c194ee8ab605c1e42e4adcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9594d5f86c194ee8ab605c1e42e4adcf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45563 } }
I20260924 04:14:34.920379 9686 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9594d5f86c194ee8ab605c1e42e4adcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:14:34.920413 9686 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9594d5f86c194ee8ab605c1e42e4adcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9594d5f86c194ee8ab605c1e42e4adcf, State: Initialized, Role: FOLLOWER
I20260924 04:14:34.920765 9686 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9594d5f86c194ee8ab605c1e42e4adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9594d5f86c194ee8ab605c1e42e4adcf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45563 } }
I20260924 04:14:34.920871 9686 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9594d5f86c194ee8ab605c1e42e4adcf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:14:34.920912 9686 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9594d5f86c194ee8ab605c1e42e4adcf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:14:34.920977 9686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9594d5f86c194ee8ab605c1e42e4adcf [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:14:34.921784 9686 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9594d5f86c194ee8ab605c1e42e4adcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9594d5f86c194ee8ab605c1e42e4adcf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45563 } }
I20260924 04:14:34.922068 9686 leader_election.cc:302] T 00000000000000000000000000000000 P 9594d5f86c194ee8ab605c1e42e4adcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9594d5f86c194ee8ab605c1e42e4adcf; no voters:
I20260924 04:14:34.922317 9686 leader_election.cc:288] T 00000000000000000000000000000000 P 9594d5f86c194ee8ab605c1e42e4adcf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:14:34.922431 9691 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9594d5f86c194ee8ab605c1e42e4adcf [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:14:34.922637 9691 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9594d5f86c194ee8ab605c1e42e4adcf [term 1 LEADER]: Becoming Leader. State: Replica: 9594d5f86c194ee8ab605c1e42e4adcf, State: Running, Role: LEADER
I20260924 04:14:34.922995 9691 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9594d5f86c194ee8ab605c1e42e4adcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9594d5f86c194ee8ab605c1e42e4adcf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45563 } }
I20260924 04:14:34.923457 9686 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9594d5f86c194ee8ab605c1e42e4adcf [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:14:34.924499 9692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9594d5f86c194ee8ab605c1e42e4adcf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9594d5f86c194ee8ab605c1e42e4adcf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9594d5f86c194ee8ab605c1e42e4adcf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45563 } } }
I20260924 04:14:34.924623 9692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9594d5f86c194ee8ab605c1e42e4adcf [sys.catalog]: This master's current role is: LEADER
I20260924 04:14:34.924913 9693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9594d5f86c194ee8ab605c1e42e4adcf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9594d5f86c194ee8ab605c1e42e4adcf. Latest consensus state: current_term: 1 leader_uuid: "9594d5f86c194ee8ab605c1e42e4adcf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9594d5f86c194ee8ab605c1e42e4adcf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45563 } } }
I20260924 04:14:34.925015 9693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9594d5f86c194ee8ab605c1e42e4adcf [sys.catalog]: This master's current role is: LEADER
I20260924 04:14:34.925391 9700 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:14:34.928309 9700 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:14:34.932909 9700 catalog_manager.cc:1347] Generated new cluster ID: 7692d458f3e24612bb0f288824773ead
I20260924 04:14:34.932974 9700 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:14:34.953135 9700 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:14:34.954257 9700 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:14:34.961795 9700 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9594d5f86c194ee8ab605c1e42e4adcf: Generated new TSK 0
I20260924 04:14:34.962531 9700 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:14:34.965399 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45563
--builtin_ntp_servers=127.0.98.20:42909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 04:14:35.075590 9710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:14:35.075866 9710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:14:35.075918 9710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:14:35.079873 9710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:14:35.079931 9710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:14:35.080008 9710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:14:35.084698 9710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:14:35.085759 9710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:14:35.087154 9710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:14:35.096064 9718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:35.096062 9715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:35.096078 9716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:35.096156 9710 server_base.cc:1034] running on GCE node
I20260924 04:14:35.096680 9710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:14:35.097273 9710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:14:35.098423 9710 hybrid_clock.cc:648] HybridClock initialized: now 1790223275098404 us; error 34 us; skew 500 ppm
I20260924 04:14:35.100135 9710 webserver.cc:460] Webserver started at http://127.0.98.1:37381/ using document root <none> and password file <none>
I20260924 04:14:35.100694 9710 fs_manager.cc:359] Metadata directory not provided
I20260924 04:14:35.100752 9710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:14:35.100943 9710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:14:35.102896 9710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/ts-0/data/instance:
uuid: "34eb5e8d5b6747ee81666dca5c733eec"
format_stamp: "Formatted at 2026-09-24 04:14:35 on dist-test-slave-p401"
server_key: "95e6ac70605e6c0be18ab3ff2cc32254"
server_key_iv: "c7f30ae8c809182e54aea0ec7210a390"
server_key_version: "encryptionkey@0"
I20260924 04:14:35.103430 9710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/ts-0/wal/instance:
uuid: "34eb5e8d5b6747ee81666dca5c733eec"
format_stamp: "Formatted at 2026-09-24 04:14:35 on dist-test-slave-p401"
server_key: "95e6ac70605e6c0be18ab3ff2cc32254"
server_key_iv: "c7f30ae8c809182e54aea0ec7210a390"
server_key_version: "encryptionkey@0"
I20260924 04:14:35.106976 9710 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:14:35.109557 9724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:35.110796 9710 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 04:14:35.110924 9710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/ts-0/wal
uuid: "34eb5e8d5b6747ee81666dca5c733eec"
format_stamp: "Formatted at 2026-09-24 04:14:35 on dist-test-slave-p401"
server_key: "95e6ac70605e6c0be18ab3ff2cc32254"
server_key_iv: "c7f30ae8c809182e54aea0ec7210a390"
server_key_version: "encryptionkey@0"
I20260924 04:14:35.111027 9710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:14:35.129601 9710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:14:35.130332 9710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:14:35.130503 9710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:14:35.131083 9710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:14:35.132076 9710 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:14:35.132124 9710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:35.132177 9710 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:14:35.132206 9710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:35.140182 9710 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40433
I20260924 04:14:35.140252 9837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40433 every 8 connection(s)
I20260924 04:14:35.141073 9710 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:14:35.141477 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 9710
I20260924 04:14:35.141592 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:14:35.141896 392 external_mini_cluster.cc:1428] Setting key bfcc865a4a744621cba099d506e9087e
I20260924 04:14:35.143821 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45563
--builtin_ntp_servers=127.0.98.20:42909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 04:14:35.150874 9689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:35.142868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48103 (local address 127.0.98.62:45563)
0924 04:14:35.142991 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:35.142996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:35.144264 (+ 1268us) server_negotiation.cc:407] Connection header received
0924 04:14:35.145876 (+ 1612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:35.145879 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:35.145929 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:35.146020 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:35.146875 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:35.147724 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:35.148868 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:35.149195 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:35.149306 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:35.150400 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:35.150404 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:35.150407 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:14:35.150423 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:35.150446 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:14:35.150456 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:35.150534 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:35.150645 (+ 111us) server_negotiation.cc:299] Negotiation successful
0924 04:14:35.150708 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 04:14:35.151744 9840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:35.143317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45563 (local address 127.0.98.1:48103)
0924 04:14:35.144096 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:35.144161 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:35.145663 (+ 1502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:35.146153 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:35.146159 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:35.146247 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:35.146726 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:35.146738 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:35.147846 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:35.147849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:35.148755 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:35.148765 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:35.149367 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:35.149371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:35.149520 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:35.150144 (+ 624us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:14:35.150146 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:35.150187 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:14:35.150251 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:14:35.150531 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:14:35.150536 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:14:35.150539 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 04:14:35.150625 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 04:14:35.150870 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":110,"threads_started":1}
I20260924 04:14:35.152844 9838 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45563
I20260924 04:14:35.153095 9838 heartbeater.cc:464] Registering TS with master...
I20260924 04:14:35.153698 9838 heartbeater.cc:511] Master 127.0.98.62:45563 requested a full tablet report, sending...
I20260924 04:14:35.155143 9650 ts_manager.cc:194] Registered new tserver with Master: 34eb5e8d5b6747ee81666dca5c733eec (127.0.98.1:40433)
I20260924 04:14:35.156288 9650 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:48103
W20260924 04:14:35.253902 9841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:14:35.254186 9841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:14:35.254225 9841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:14:35.258395 9841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:14:35.258476 9841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:14:35.258533 9841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:14:35.263535 9841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:14:35.264706 9841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:14:35.266135 9841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:14:35.275763 9846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:35.275744 9849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:35.275794 9847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:35.276037 9841 server_base.cc:1034] running on GCE node
I20260924 04:14:35.276440 9841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:14:35.277069 9841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:14:35.278251 9841 hybrid_clock.cc:648] HybridClock initialized: now 1790223275278223 us; error 36 us; skew 500 ppm
I20260924 04:14:35.280332 9841 webserver.cc:460] Webserver started at http://127.0.98.2:33467/ using document root <none> and password file <none>
I20260924 04:14:35.280915 9841 fs_manager.cc:359] Metadata directory not provided
I20260924 04:14:35.280984 9841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:14:35.281208 9841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:14:35.283402 9841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/ts-1/data/instance:
uuid: "3446b56aa53c43daad300bfe3be114de"
format_stamp: "Formatted at 2026-09-24 04:14:35 on dist-test-slave-p401"
server_key: "4459baff9afaca50638fe1115100c29a"
server_key_iv: "99b1dd28426b480e68d697b550cf5080"
server_key_version: "encryptionkey@0"
I20260924 04:14:35.283970 9841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3446b56aa53c43daad300bfe3be114de"
format_stamp: "Formatted at 2026-09-24 04:14:35 on dist-test-slave-p401"
server_key: "4459baff9afaca50638fe1115100c29a"
server_key_iv: "99b1dd28426b480e68d697b550cf5080"
server_key_version: "encryptionkey@0"
I20260924 04:14:35.287905 9841 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 04:14:35.290499 9855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:35.291869 9841 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:14:35.291973 9841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/ts-1/wal
uuid: "3446b56aa53c43daad300bfe3be114de"
format_stamp: "Formatted at 2026-09-24 04:14:35 on dist-test-slave-p401"
server_key: "4459baff9afaca50638fe1115100c29a"
server_key_iv: "99b1dd28426b480e68d697b550cf5080"
server_key_version: "encryptionkey@0"
I20260924 04:14:35.292097 9841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:14:35.319383 9841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:14:35.320125 9841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:14:35.320310 9841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:14:35.321027 9841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:14:35.322175 9841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:14:35.322228 9841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:35.322270 9841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:14:35.322284 9841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:35.331115 9841 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45537
I20260924 04:14:35.331146 9968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45537 every 8 connection(s)
I20260924 04:14:35.332113 9841 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:14:35.340273 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 9841
I20260924 04:14:35.340399 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:14:35.340672 392 external_mini_cluster.cc:1428] Setting key 6e7390d5b0d0e07a49a5cb3b7b2ae8b0
I20260924 04:14:35.341253 9689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:35.334021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54915 (local address 127.0.98.62:45563)
0924 04:14:35.334175 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:35.334179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:35.335315 (+ 1136us) server_negotiation.cc:407] Connection header received
0924 04:14:35.336734 (+ 1419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:35.336737 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:35.336793 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:35.336866 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:35.337775 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:35.338291 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:35.339423 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:35.339646 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:35.339744 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:35.340792 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:35.340795 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:35.340798 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:14:35.340811 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:35.340832 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:14:35.340842 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:35.340899 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:35.341049 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 04:14:35.341109 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 04:14:35.342247 9971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:35.334430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45563 (local address 127.0.98.2:54915)
0924 04:14:35.335135 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:35.335177 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:35.336495 (+ 1318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:35.337047 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:35.337054 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:35.337150 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:35.337638 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:35.337650 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:35.338403 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:35.338405 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:35.339309 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:35.339318 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:35.339786 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:35.339790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:35.339946 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:35.340530 (+ 584us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:14:35.340531 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:35.340573 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:14:35.340642 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:14:35.340912 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:14:35.340926 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:14:35.340932 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 04:14:35.341038 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 04:14:35.341340 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":115,"threads_started":1}
I20260924 04:14:35.342779 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45563
--builtin_ntp_servers=127.0.98.20:42909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 04:14:35.343343 9969 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45563
I20260924 04:14:35.343592 9969 heartbeater.cc:464] Registering TS with master...
I20260924 04:14:35.344241 9969 heartbeater.cc:511] Master 127.0.98.62:45563 requested a full tablet report, sending...
I20260924 04:14:35.345301 9650 ts_manager.cc:194] Registered new tserver with Master: 3446b56aa53c43daad300bfe3be114de (127.0.98.2:45537)
I20260924 04:14:35.345853 9650 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:54915
W20260924 04:14:35.455624 9972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:14:35.455916 9972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:14:35.455969 9972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:14:35.460073 9972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:14:35.460137 9972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:14:35.460218 9972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:14:35.465263 9972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:14:35.466357 9972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:14:35.467793 9972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:14:35.477301 9977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:35.477433 9972 server_base.cc:1034] running on GCE node
W20260924 04:14:35.477327 9980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:35.477311 9978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:35.477949 9972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:14:35.478554 9972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:14:35.479715 9972 hybrid_clock.cc:648] HybridClock initialized: now 1790223275479687 us; error 38 us; skew 500 ppm
I20260924 04:14:35.481374 9972 webserver.cc:460] Webserver started at http://127.0.98.3:40185/ using document root <none> and password file <none>
I20260924 04:14:35.482002 9972 fs_manager.cc:359] Metadata directory not provided
I20260924 04:14:35.482062 9972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:14:35.482237 9972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:14:35.484236 9972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/ts-2/data/instance:
uuid: "9225389ee2e349609737773365f86f41"
format_stamp: "Formatted at 2026-09-24 04:14:35 on dist-test-slave-p401"
server_key: "0e819c0ab3bd042cb329c625a5b8ef98"
server_key_iv: "e10449d4dda807c067f3c3c3378d48f1"
server_key_version: "encryptionkey@0"
I20260924 04:14:35.484747 9972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9225389ee2e349609737773365f86f41"
format_stamp: "Formatted at 2026-09-24 04:14:35 on dist-test-slave-p401"
server_key: "0e819c0ab3bd042cb329c625a5b8ef98"
server_key_iv: "e10449d4dda807c067f3c3c3378d48f1"
server_key_version: "encryptionkey@0"
I20260924 04:14:35.488260 9972 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:14:35.490669 9986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:35.491911 9972 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:14:35.492007 9972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/ts-2/wal
uuid: "9225389ee2e349609737773365f86f41"
format_stamp: "Formatted at 2026-09-24 04:14:35 on dist-test-slave-p401"
server_key: "0e819c0ab3bd042cb329c625a5b8ef98"
server_key_iv: "e10449d4dda807c067f3c3c3378d48f1"
server_key_version: "encryptionkey@0"
I20260924 04:14:35.492108 9972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:14:35.514592 9972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:14:35.515329 9972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:14:35.515502 9972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:14:35.516091 9972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:14:35.517025 9972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:14:35.517069 9972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:35.517115 9972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:14:35.517143 9972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:35.525554 9972 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42663
I20260924 04:14:35.525600 10099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42663 every 8 connection(s)
I20260924 04:14:35.526504 9972 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:14:35.529194 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 9972
I20260924 04:14:35.529307 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790223216265354-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:14:35.529594 392 external_mini_cluster.cc:1428] Setting key 24abb62099972e069903ec0f8f92c5b2
I20260924 04:14:35.536165 9689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:35.528366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39111 (local address 127.0.98.62:45563)
0924 04:14:35.528508 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:35.528512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:35.529804 (+ 1292us) server_negotiation.cc:407] Connection header received
0924 04:14:35.531583 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:35.531587 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:35.531641 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:35.531715 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:35.532641 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:35.533173 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:35.534279 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:35.534487 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:35.534561 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:35.535700 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:35.535704 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:35.535706 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:14:35.535720 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:35.535741 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:14:35.535751 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:35.535829 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:35.535972 (+ 143us) server_negotiation.cc:299] Negotiation successful
0924 04:14:35.536026 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 04:14:35.537015 10102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:35.528878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45563 (local address 127.0.98.3:39111)
0924 04:14:35.529630 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:35.529688 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:35.531257 (+ 1569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:35.531907 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:35.531914 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:35.532006 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:35.532512 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:35.532525 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:35.533279 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:35.533282 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:35.534184 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:35.534192 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:35.534610 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:35.534613 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:35.534751 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:35.535372 (+ 621us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:14:35.535373 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:35.535415 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:14:35.535484 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:14:35.535828 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:14:35.535833 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:14:35.535836 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 04:14:35.535923 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 04:14:35.536157 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":146,"threads_started":1}
I20260924 04:14:35.538192 10100 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45563
I20260924 04:14:35.538439 10100 heartbeater.cc:464] Registering TS with master...
I20260924 04:14:35.538949 10100 heartbeater.cc:511] Master 127.0.98.62:45563 requested a full tablet report, sending...
I20260924 04:14:35.539948 9650 ts_manager.cc:194] Registered new tserver with Master: 9225389ee2e349609737773365f86f41 (127.0.98.3:42663)
I20260924 04:14:35.540509 9650 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:39111
I20260924 04:14:35.543548 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 04:14:35.546286 9689 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:35.545393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35846 (local address 127.0.98.62:45563)
0924 04:14:35.545512 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:35.545516 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:35.545650 (+ 134us) server_negotiation.cc:407] Connection header received
0924 04:14:35.545778 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:35.545781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:35.545825 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:35.545886 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:35.546131 (+ 245us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35846: BlockingRecv error: recv got EOF from 127.0.0.1:35846 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 04:14:35.546352 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 9710
I20260924 04:14:35.551384 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 9841
I20260924 04:14:35.555546 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 9972
I20260924 04:14:35.559777 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 9619
2026-09-24T04:14:35Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (848 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-24T04:14:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:14:35Z Disabled control of system clock
I20260924 04:14:35.586293 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38099
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38099
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 04:14:35.694901 10111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:14:35.695190 10111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:14:35.695257 10111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:14:35.699321 10111 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:14:35.699393 10111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:14:35.699406 10111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:14:35.699422 10111 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:14:35.699435 10111 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:14:35.704308 10111 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38099
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:14:35.705380 10111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:14:35.706717 10111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:14:35.714879 10119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:35.714828 10116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:35.714831 10117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:35.715250 10111 server_base.cc:1034] running on GCE node
I20260924 04:14:35.715808 10111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:14:35.716866 10111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:14:35.718046 10111 hybrid_clock.cc:648] HybridClock initialized: now 1790223275718019 us; error 39 us; skew 500 ppm
I20260924 04:14:35.719926 10111 webserver.cc:460] Webserver started at http://127.0.98.62:37209/ using document root <none> and password file <none>
I20260924 04:14:35.720470 10111 fs_manager.cc:359] Metadata directory not provided
I20260924 04:14:35.720530 10111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:14:35.720769 10111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:14:35.722846 10111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/master-0/data/instance:
uuid: "8057e98ea860406793bb578261e45524"
format_stamp: "Formatted at 2026-09-24 04:14:35 on dist-test-slave-p401"
server_key: "8efa6ba1fb47c8bea38682afe12828dd"
server_key_iv: "1a0dc86eaa90bb6f1a7c3a2a152d6363"
server_key_version: "encryptionkey@0"
I20260924 04:14:35.723419 10111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/master-0/wal/instance:
uuid: "8057e98ea860406793bb578261e45524"
format_stamp: "Formatted at 2026-09-24 04:14:35 on dist-test-slave-p401"
server_key: "8efa6ba1fb47c8bea38682afe12828dd"
server_key_iv: "1a0dc86eaa90bb6f1a7c3a2a152d6363"
server_key_version: "encryptionkey@0"
I20260924 04:14:35.727104 10111 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:14:35.729579 10125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:35.730767 10111 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:14:35.730880 10111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/master-0/wal
uuid: "8057e98ea860406793bb578261e45524"
format_stamp: "Formatted at 2026-09-24 04:14:35 on dist-test-slave-p401"
server_key: "8efa6ba1fb47c8bea38682afe12828dd"
server_key_iv: "1a0dc86eaa90bb6f1a7c3a2a152d6363"
server_key_version: "encryptionkey@0"
I20260924 04:14:35.730983 10111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:14:35.735575 10111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:14:35.736191 10111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:14:35.736346 10111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:14:35.742942 10111 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38099
I20260924 04:14:35.742956 10177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38099 every 8 connection(s)
I20260924 04:14:35.743950 10111 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:14:35.747047 10178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:14:35.751888 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 10111
I20260924 04:14:35.752007 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/master-0/wal/instance
I20260924 04:14:35.752249 392 external_mini_cluster.cc:1428] Setting key a4d0418bd16de29489aca885cb0202f7
I20260924 04:14:35.752746 10178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8057e98ea860406793bb578261e45524: Bootstrap starting.
I20260924 04:14:35.755043 10178 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8057e98ea860406793bb578261e45524: Neither blocks nor log segments found. Creating new log.
I20260924 04:14:35.755820 10178 log.cc:824] T 00000000000000000000000000000000 P 8057e98ea860406793bb578261e45524: Log is configured to *not* fsync() on all Append() calls
I20260924 04:14:35.757720 10181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:35.753661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34646 (local address 127.0.98.62:38099)
0924 04:14:35.754306 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:35.754322 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:35.754359 (+ 37us) server_negotiation.cc:407] Connection header received
0924 04:14:35.754990 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:35.755008 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:35.755433 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:35.755716 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:35.756475 (+ 759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:35.756481 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:35.756497 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:14:35.756508 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:35.756576 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:14:35.756599 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:35.756700 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:35.756802 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 04:14:35.757030 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":129,"threads_started":1}
I20260924 04:14:35.758188 10178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8057e98ea860406793bb578261e45524: No bootstrap required, opened a new log
I20260924 04:14:35.762059 10178 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8057e98ea860406793bb578261e45524 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8057e98ea860406793bb578261e45524" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38099 } }
I20260924 04:14:35.762342 10178 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8057e98ea860406793bb578261e45524 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:14:35.762391 10178 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8057e98ea860406793bb578261e45524 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8057e98ea860406793bb578261e45524, State: Initialized, Role: FOLLOWER
I20260924 04:14:35.762850 10178 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8057e98ea860406793bb578261e45524 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8057e98ea860406793bb578261e45524" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38099 } }
I20260924 04:14:35.762979 10178 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8057e98ea860406793bb578261e45524 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:14:35.763041 10178 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8057e98ea860406793bb578261e45524 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:14:35.763134 10178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8057e98ea860406793bb578261e45524 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:14:35.764344 10178 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8057e98ea860406793bb578261e45524 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8057e98ea860406793bb578261e45524" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38099 } }
I20260924 04:14:35.764686 10178 leader_election.cc:302] T 00000000000000000000000000000000 P 8057e98ea860406793bb578261e45524 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8057e98ea860406793bb578261e45524; no voters:
I20260924 04:14:35.764982 10178 leader_election.cc:288] T 00000000000000000000000000000000 P 8057e98ea860406793bb578261e45524 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:14:35.765130 10183 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8057e98ea860406793bb578261e45524 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:14:35.765380 10183 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8057e98ea860406793bb578261e45524 [term 1 LEADER]: Becoming Leader. State: Replica: 8057e98ea860406793bb578261e45524, State: Running, Role: LEADER
I20260924 04:14:35.765684 10183 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8057e98ea860406793bb578261e45524 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8057e98ea860406793bb578261e45524" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38099 } }
I20260924 04:14:35.766204 10178 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8057e98ea860406793bb578261e45524 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:14:35.767349 10184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8057e98ea860406793bb578261e45524 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8057e98ea860406793bb578261e45524" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8057e98ea860406793bb578261e45524" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38099 } } }
I20260924 04:14:35.767374 10185 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8057e98ea860406793bb578261e45524 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8057e98ea860406793bb578261e45524. Latest consensus state: current_term: 1 leader_uuid: "8057e98ea860406793bb578261e45524" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8057e98ea860406793bb578261e45524" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38099 } } }
I20260924 04:14:35.767500 10184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8057e98ea860406793bb578261e45524 [sys.catalog]: This master's current role is: LEADER
I20260924 04:14:35.767503 10185 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8057e98ea860406793bb578261e45524 [sys.catalog]: This master's current role is: LEADER
I20260924 04:14:35.767823 10192 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:14:35.770792 10192 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:14:35.776461 10192 catalog_manager.cc:1347] Generated new cluster ID: 853a321b25464ef0a7477d25451021b3
I20260924 04:14:35.776546 10192 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:14:35.804592 10192 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:14:35.804972 10192 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:14:35.814263 10192 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8057e98ea860406793bb578261e45524: Generated new TSK 0
I20260924 04:14:35.815145 10192 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:14:35.821974 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38099
--builtin_ntp_servers=127.0.98.20:33095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 04:14:35.931788 10202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:14:35.932049 10202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:14:35.932088 10202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:14:35.936268 10202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:14:35.936335 10202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:14:35.936393 10202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:14:35.941196 10202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:14:35.942292 10202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:14:35.943703 10202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:14:35.953332 10207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:35.953332 10210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:35.953354 10208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:35.953740 10202 server_base.cc:1034] running on GCE node
I20260924 04:14:35.954150 10202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:14:35.954802 10202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:14:35.955963 10202 hybrid_clock.cc:648] HybridClock initialized: now 1790223275955937 us; error 36 us; skew 500 ppm
I20260924 04:14:35.957896 10202 webserver.cc:460] Webserver started at http://127.0.98.1:44807/ using document root <none> and password file <none>
I20260924 04:14:35.958458 10202 fs_manager.cc:359] Metadata directory not provided
I20260924 04:14:35.958523 10202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:14:35.958734 10202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:14:35.960827 10202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/ts-0/data/instance:
uuid: "6b89467068c041d3b83e9108abcaa954"
format_stamp: "Formatted at 2026-09-24 04:14:35 on dist-test-slave-p401"
server_key: "529287fa1e20e2dcc0de0311e8e987f4"
server_key_iv: "0e2a36e977149df995b74d9d70cffe0f"
server_key_version: "encryptionkey@0"
I20260924 04:14:35.961356 10202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6b89467068c041d3b83e9108abcaa954"
format_stamp: "Formatted at 2026-09-24 04:14:35 on dist-test-slave-p401"
server_key: "529287fa1e20e2dcc0de0311e8e987f4"
server_key_iv: "0e2a36e977149df995b74d9d70cffe0f"
server_key_version: "encryptionkey@0"
I20260924 04:14:35.965041 10202 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260924 04:14:35.967602 10216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:35.968866 10202 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:14:35.968971 10202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/ts-0/wal
uuid: "6b89467068c041d3b83e9108abcaa954"
format_stamp: "Formatted at 2026-09-24 04:14:35 on dist-test-slave-p401"
server_key: "529287fa1e20e2dcc0de0311e8e987f4"
server_key_iv: "0e2a36e977149df995b74d9d70cffe0f"
server_key_version: "encryptionkey@0"
I20260924 04:14:35.969070 10202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:14:35.973774 10202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:14:35.974367 10202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:14:35.974522 10202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:14:35.975041 10202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:14:35.975944 10202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:14:35.975991 10202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:35.976040 10202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:14:35.976069 10202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:35.984525 10202 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44991
I20260924 04:14:35.984546 10329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44991 every 8 connection(s)
I20260924 04:14:35.985461 10202 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:14:35.987746 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 10202
I20260924 04:14:35.987850 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:14:35.988155 392 external_mini_cluster.cc:1428] Setting key 78b8add0340ac8f6eaf4293bc2c3adde
I20260924 04:14:35.990166 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38099
--builtin_ntp_servers=127.0.98.20:33095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 04:14:35.992051 10181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:35.987300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58065 (local address 127.0.98.62:38099)
0924 04:14:35.987419 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:35.987423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:35.988750 (+ 1327us) server_negotiation.cc:407] Connection header received
0924 04:14:35.990275 (+ 1525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:35.990279 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:35.990325 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:35.990409 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:35.991639 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:35.991642 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:35.991645 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:14:35.991658 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:35.991682 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:14:35.991691 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:35.991788 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:35.991876 (+ 88us) server_negotiation.cc:299] Negotiation successful
0924 04:14:35.991918 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 04:14:35.992889 10332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:35.987782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38099 (local address 127.0.98.1:58065)
0924 04:14:35.988597 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:35.988643 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:35.990021 (+ 1378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:35.990597 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:35.990605 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:35.990730 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:35.991297 (+ 567us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:14:35.991299 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:35.991370 (+ 71us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:14:35.991470 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:14:35.991771 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:14:35.991776 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:14:35.991779 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 04:14:35.991904 (+ 125us) client_negotiation.cc:240] Negotiation successful
0924 04:14:35.992102 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":135,"threads_started":1}
I20260924 04:14:35.994125 10330 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38099
I20260924 04:14:35.994371 10330 heartbeater.cc:464] Registering TS with master...
I20260924 04:14:35.994777 10330 heartbeater.cc:511] Master 127.0.98.62:38099 requested a full tablet report, sending...
I20260924 04:14:35.996284 10142 ts_manager.cc:194] Registered new tserver with Master: 6b89467068c041d3b83e9108abcaa954 (127.0.98.1:44991)
W20260924 04:14:36.100641 10333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:14:36.100920 10333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:14:36.100960 10333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:14:36.105063 10333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:14:36.105131 10333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:14:36.105195 10333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:14:36.109966 10333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:14:36.111084 10333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:14:36.112541 10333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:14:36.122125 10341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:36.122273 10338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:36.122349 10333 server_base.cc:1034] running on GCE node
W20260924 04:14:36.122212 10339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:36.122989 10333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:14:36.123679 10333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:14:36.124837 10333 hybrid_clock.cc:648] HybridClock initialized: now 1790223276124818 us; error 35 us; skew 500 ppm
I20260924 04:14:36.126762 10333 webserver.cc:460] Webserver started at http://127.0.98.2:45133/ using document root <none> and password file <none>
I20260924 04:14:36.127370 10333 fs_manager.cc:359] Metadata directory not provided
I20260924 04:14:36.127439 10333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:14:36.127637 10333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:14:36.129787 10333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/ts-1/data/instance:
uuid: "054d55c4651a45888440a0f1f5a6cb36"
format_stamp: "Formatted at 2026-09-24 04:14:36 on dist-test-slave-p401"
server_key: "43e4b1fef6ab797e8bba01d9d13fff98"
server_key_iv: "795e8b1e629ce55dcec834fbc0a697f7"
server_key_version: "encryptionkey@0"
I20260924 04:14:36.130314 10333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/ts-1/wal/instance:
uuid: "054d55c4651a45888440a0f1f5a6cb36"
format_stamp: "Formatted at 2026-09-24 04:14:36 on dist-test-slave-p401"
server_key: "43e4b1fef6ab797e8bba01d9d13fff98"
server_key_iv: "795e8b1e629ce55dcec834fbc0a697f7"
server_key_version: "encryptionkey@0"
I20260924 04:14:36.133999 10333 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260924 04:14:36.136420 10347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:36.137609 10333 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:14:36.137720 10333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/ts-1/wal
uuid: "054d55c4651a45888440a0f1f5a6cb36"
format_stamp: "Formatted at 2026-09-24 04:14:36 on dist-test-slave-p401"
server_key: "43e4b1fef6ab797e8bba01d9d13fff98"
server_key_iv: "795e8b1e629ce55dcec834fbc0a697f7"
server_key_version: "encryptionkey@0"
I20260924 04:14:36.137826 10333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:14:36.142323 10333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:14:36.142913 10333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:14:36.143067 10333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:14:36.143611 10333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:14:36.144475 10333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:14:36.144520 10333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:36.144569 10333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:14:36.144598 10333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:36.152665 10333 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41435
I20260924 04:14:36.152694 10460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41435 every 8 connection(s)
I20260924 04:14:36.153538 10333 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:14:36.156388 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 10333
I20260924 04:14:36.156508 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:14:36.156848 392 external_mini_cluster.cc:1428] Setting key 69ce9bd4dc815354a1902bf3fb15d5b2
I20260924 04:14:36.158746 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38099
--builtin_ntp_servers=127.0.98.20:33095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 04:14:36.159695 10181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:36.155270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41757 (local address 127.0.98.62:38099)
0924 04:14:36.155415 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:36.155420 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:36.156592 (+ 1172us) server_negotiation.cc:407] Connection header received
0924 04:14:36.158160 (+ 1568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:36.158163 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:36.158197 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:36.158258 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:36.159285 (+ 1027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:36.159288 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:36.159290 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:14:36.159300 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:36.159318 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:14:36.159326 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:36.159395 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:36.159516 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 04:14:36.159557 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:14:36.160627 10463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:36.155675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38099 (local address 127.0.98.2:41757)
0924 04:14:36.156428 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:36.156485 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:36.157926 (+ 1441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:36.158413 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:36.158420 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:36.158506 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:36.159009 (+ 503us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:14:36.159010 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:36.159051 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:14:36.159118 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:14:36.159392 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:14:36.159400 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:14:36.159405 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 04:14:36.159552 (+ 147us) client_negotiation.cc:240] Negotiation successful
0924 04:14:36.159926 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":124,"threads_started":1}
I20260924 04:14:36.161590 10461 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38099
I20260924 04:14:36.161823 10461 heartbeater.cc:464] Registering TS with master...
I20260924 04:14:36.162253 10461 heartbeater.cc:511] Master 127.0.98.62:38099 requested a full tablet report, sending...
I20260924 04:14:36.163311 10142 ts_manager.cc:194] Registered new tserver with Master: 054d55c4651a45888440a0f1f5a6cb36 (127.0.98.2:41435)
W20260924 04:14:36.271092 10464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:14:36.271461 10464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:14:36.271517 10464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:14:36.275683 10464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:14:36.275739 10464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:14:36.275806 10464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:14:36.280588 10464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:14:36.281615 10464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:14:36.282945 10464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:14:36.291980 10469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:36.291982 10472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:36.291980 10470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:36.292348 10464 server_base.cc:1034] running on GCE node
I20260924 04:14:36.292704 10464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:14:36.293289 10464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:14:36.294446 10464 hybrid_clock.cc:648] HybridClock initialized: now 1790223276294418 us; error 43 us; skew 500 ppm
I20260924 04:14:36.296144 10464 webserver.cc:460] Webserver started at http://127.0.98.3:39129/ using document root <none> and password file <none>
I20260924 04:14:36.296697 10464 fs_manager.cc:359] Metadata directory not provided
I20260924 04:14:36.296763 10464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:14:36.296958 10464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:14:36.298938 10464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/ts-2/data/instance:
uuid: "d8f19978de9240d2bdf7247552e064d4"
format_stamp: "Formatted at 2026-09-24 04:14:36 on dist-test-slave-p401"
server_key: "b779de5abd1d6848143795d0625cd45b"
server_key_iv: "222cc4fc14b365d339ebafc77cf6f9d5"
server_key_version: "encryptionkey@0"
I20260924 04:14:36.299482 10464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d8f19978de9240d2bdf7247552e064d4"
format_stamp: "Formatted at 2026-09-24 04:14:36 on dist-test-slave-p401"
server_key: "b779de5abd1d6848143795d0625cd45b"
server_key_iv: "222cc4fc14b365d339ebafc77cf6f9d5"
server_key_version: "encryptionkey@0"
I20260924 04:14:36.302855 10464 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260924 04:14:36.305097 10478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:36.306213 10464 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:14:36.306329 10464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/ts-2/wal
uuid: "d8f19978de9240d2bdf7247552e064d4"
format_stamp: "Formatted at 2026-09-24 04:14:36 on dist-test-slave-p401"
server_key: "b779de5abd1d6848143795d0625cd45b"
server_key_iv: "222cc4fc14b365d339ebafc77cf6f9d5"
server_key_version: "encryptionkey@0"
I20260924 04:14:36.306418 10464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:14:36.310834 10464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:14:36.311443 10464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:14:36.311587 10464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:14:36.312090 10464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:14:36.312943 10464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:14:36.312987 10464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:36.313035 10464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:14:36.313060 10464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:36.321079 10464 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39783
I20260924 04:14:36.321126 10591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39783 every 8 connection(s)
I20260924 04:14:36.321940 10464 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:14:36.324674 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 10464
I20260924 04:14:36.324787 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790223216265354-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:14:36.325071 392 external_mini_cluster.cc:1428] Setting key 9d53f470973742623e1dbffa4876fe71
I20260924 04:14:36.327913 10181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:36.323654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57539 (local address 127.0.98.62:38099)
0924 04:14:36.323785 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:36.323788 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:36.324844 (+ 1056us) server_negotiation.cc:407] Connection header received
0924 04:14:36.326292 (+ 1448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:36.326294 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:36.326331 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:36.326393 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:36.327551 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:36.327553 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:36.327555 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:14:36.327564 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:36.327583 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:14:36.327590 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:36.327655 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:36.327787 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 04:14:36.327818 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 04:14:36.328840 10594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:36.324064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38099 (local address 127.0.98.3:57539)
0924 04:14:36.324674 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:36.324742 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:36.326090 (+ 1348us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:36.326534 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:36.326543 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:36.326659 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:36.327321 (+ 662us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:14:36.327323 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:36.327374 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:14:36.327443 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:14:36.327679 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:14:36.327684 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:14:36.327688 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 04:14:36.327829 (+ 141us) client_negotiation.cc:240] Negotiation successful
0924 04:14:36.328039 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":104,"threads_started":1}
I20260924 04:14:36.329794 10592 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38099
I20260924 04:14:36.329999 10592 heartbeater.cc:464] Registering TS with master...
I20260924 04:14:36.330358 10592 heartbeater.cc:511] Master 127.0.98.62:38099 requested a full tablet report, sending...
I20260924 04:14:36.331306 10142 ts_manager.cc:194] Registered new tserver with Master: d8f19978de9240d2bdf7247552e064d4 (127.0.98.3:39783)
I20260924 04:14:36.338949 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 04:14:36.341511 10181 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:14:36.340709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34650 (local address 127.0.98.62:38099)
0924 04:14:36.340813 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:36.340817 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:36.341037 (+ 220us) server_negotiation.cc:407] Connection header received
0924 04:14:36.341131 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:36.341133 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:36.341162 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:36.341226 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:36.341393 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34650: BlockingRecv error: recv got EOF from 127.0.0.1:34650 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 04:14:36.341766 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 10202
I20260924 04:14:36.346531 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 10333
I20260924 04:14:36.350961 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 10464
I20260924 04:14:36.355271 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 10111
2026-09-24T04:14:36Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (795 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:14:36 dist-test-slave-p401 krb5kdc[10602](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:14:36 dist-test-slave-p401 krb5kdc[10602](info): set up 2 sockets
Sep 24 04:14:36 dist-test-slave-p401 krb5kdc[10602](info): commencing operation
krb5kdc: starting...
W20260924 04:14:37.315868 392 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/krb5kdc/test-user.keytab.
Sep 24 04:14:37 dist-test-slave-p401 krb5kdc[10602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223277, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:14:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:14:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:14:37.435066 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44493
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44493
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:14:37.544996 10615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:14:37.545279 10615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:14:37.545320 10615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:14:37.549382 10615 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:14:37.549448 10615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:14:37.549466 10615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:14:37.549481 10615 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:14:37.549495 10615 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:14:37.555056 10615 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:14:37.556186 10615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:14:37.557214 10615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:14:37.565560 10620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:37.565579 10623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:37.565744 10615 server_base.cc:1034] running on GCE node
W20260924 04:14:37.565557 10621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:37.566459 10615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:14:37.567626 10615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:14:37.568800 10615 hybrid_clock.cc:648] HybridClock initialized: now 1790223277568778 us; error 39 us; skew 500 ppm
Sep 24 04:14:37 dist-test-slave-p401 krb5kdc[10602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223277, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:14:37.571519 10615 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:14:37.572404 10615 webserver.cc:460] Webserver started at http://127.0.98.62:34299/ using document root <none> and password file <none>
I20260924 04:14:37.572925 10615 fs_manager.cc:359] Metadata directory not provided
I20260924 04:14:37.572985 10615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:14:37.573148 10615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:14:37.575186 10615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/master-0/data/instance:
uuid: "2e35a25996be41b691945ea647935748"
format_stamp: "Formatted at 2026-09-24 04:14:37 on dist-test-slave-p401"
server_key: "5f40377b86f6e1fea739e38e78528bb9"
server_key_iv: "6cd5c016b6da62e251ddee86ddf1c21b"
server_key_version: "encryptionkey@0"
I20260924 04:14:37.575773 10615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/master-0/wal/instance:
uuid: "2e35a25996be41b691945ea647935748"
format_stamp: "Formatted at 2026-09-24 04:14:37 on dist-test-slave-p401"
server_key: "5f40377b86f6e1fea739e38e78528bb9"
server_key_iv: "6cd5c016b6da62e251ddee86ddf1c21b"
server_key_version: "encryptionkey@0"
I20260924 04:14:37.579408 10615 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 04:14:37.581816 10630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:37.583101 10615 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:14:37.583206 10615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/master-0/wal
uuid: "2e35a25996be41b691945ea647935748"
format_stamp: "Formatted at 2026-09-24 04:14:37 on dist-test-slave-p401"
server_key: "5f40377b86f6e1fea739e38e78528bb9"
server_key_iv: "6cd5c016b6da62e251ddee86ddf1c21b"
server_key_version: "encryptionkey@0"
I20260924 04:14:37.583374 10615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:14:37.599937 10615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:14:37.603111 10615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:14:37.603329 10615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:14:37.609774 10615 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44493
I20260924 04:14:37.610038 10682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44493 every 8 connection(s)
I20260924 04:14:37.610716 10615 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:14:37.610918 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 10615
I20260924 04:14:37.611016 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/master-0/wal/instance
I20260924 04:14:37.611380 392 external_mini_cluster.cc:1428] Setting key 756a1d51acdccbd48d13c9a45278a193
I20260924 04:14:37.613870 10683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:14:37.619901 10683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e35a25996be41b691945ea647935748: Bootstrap starting.
Sep 24 04:14:37 dist-test-slave-p401 krb5kdc[10602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223277, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:14:37.622403 10683 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e35a25996be41b691945ea647935748: Neither blocks nor log segments found. Creating new log.
I20260924 04:14:37.623201 10683 log.cc:824] T 00000000000000000000000000000000 P 2e35a25996be41b691945ea647935748: Log is configured to *not* fsync() on all Append() calls
I20260924 04:14:37.625396 10683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e35a25996be41b691945ea647935748: No bootstrap required, opened a new log
I20260924 04:14:37.625489 10686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:37.612753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60298 (local address 127.0.98.62:44493)
0924 04:14:37.613179 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:37.613195 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:37.613232 (+ 37us) server_negotiation.cc:407] Connection header received
0924 04:14:37.613875 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:37.613891 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:37.614256 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:37.614570 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:37.615353 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:37.616828 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:37.617679 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:37.618348 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:37.618503 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:37.620774 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:37.620780 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:37.620794 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:37.620827 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:37.622623 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:37.623037 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:37.623042 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:37.623048 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:37.623182 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:37.623443 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:37.623447 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:37.623448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:37.623875 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:37.624049 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:37.624291 (+ 242us) server_negotiation.cc:299] Negotiation successful
0924 04:14:37.624585 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":137,"threads_started":1}
I20260924 04:14:37.629114 10683 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e35a25996be41b691945ea647935748 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e35a25996be41b691945ea647935748" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44493 } }
I20260924 04:14:37.629333 10683 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e35a25996be41b691945ea647935748 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:14:37.629366 10683 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e35a25996be41b691945ea647935748 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e35a25996be41b691945ea647935748, State: Initialized, Role: FOLLOWER
I20260924 04:14:37.629732 10683 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e35a25996be41b691945ea647935748 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e35a25996be41b691945ea647935748" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44493 } }
I20260924 04:14:37.629853 10683 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e35a25996be41b691945ea647935748 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:14:37.629899 10683 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e35a25996be41b691945ea647935748 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:14:37.629966 10683 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e35a25996be41b691945ea647935748 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:14:37.630787 10683 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e35a25996be41b691945ea647935748 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e35a25996be41b691945ea647935748" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44493 } }
I20260924 04:14:37.631125 10683 leader_election.cc:302] T 00000000000000000000000000000000 P 2e35a25996be41b691945ea647935748 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e35a25996be41b691945ea647935748; no voters:
I20260924 04:14:37.631441 10683 leader_election.cc:288] T 00000000000000000000000000000000 P 2e35a25996be41b691945ea647935748 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:14:37.631583 10688 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e35a25996be41b691945ea647935748 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:14:37.631803 10688 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e35a25996be41b691945ea647935748 [term 1 LEADER]: Becoming Leader. State: Replica: 2e35a25996be41b691945ea647935748, State: Running, Role: LEADER
I20260924 04:14:37.632114 10688 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e35a25996be41b691945ea647935748 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e35a25996be41b691945ea647935748" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44493 } }
I20260924 04:14:37.632599 10683 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e35a25996be41b691945ea647935748 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:14:37.633579 10689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e35a25996be41b691945ea647935748 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e35a25996be41b691945ea647935748" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e35a25996be41b691945ea647935748" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44493 } } }
I20260924 04:14:37.633687 10689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e35a25996be41b691945ea647935748 [sys.catalog]: This master's current role is: LEADER
I20260924 04:14:37.633965 10690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e35a25996be41b691945ea647935748 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e35a25996be41b691945ea647935748. Latest consensus state: current_term: 1 leader_uuid: "2e35a25996be41b691945ea647935748" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e35a25996be41b691945ea647935748" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44493 } } }
I20260924 04:14:37.634052 10690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e35a25996be41b691945ea647935748 [sys.catalog]: This master's current role is: LEADER
I20260924 04:14:37.634308 10697 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:14:37.637266 10697 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:14:37.642697 10697 catalog_manager.cc:1347] Generated new cluster ID: bdc8b9367592403d9747ef4fceedc0fe
I20260924 04:14:37.642762 10697 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:14:37.667748 10697 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:14:37.668778 10697 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:14:37.689971 10697 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e35a25996be41b691945ea647935748: Generated new TSK 0
I20260924 04:14:37.690723 10697 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:14:37.730607 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44493
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:14:37.843325 10709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:14:37.843582 10709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:14:37.843624 10709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:14:37.847635 10709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:14:37.847697 10709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:14:37.847754 10709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:14:37.852934 10709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:14:37.853948 10709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:14:37.854964 10709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:14:37.864113 10717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:37.864142 10714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:37.864112 10715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:37.864401 10709 server_base.cc:1034] running on GCE node
I20260924 04:14:37.864816 10709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:14:37.865402 10709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:14:37.866555 10709 hybrid_clock.cc:648] HybridClock initialized: now 1790223277866533 us; error 30 us; skew 500 ppm
Sep 24 04:14:37 dist-test-slave-p401 krb5kdc[10602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223277, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:14:37.869165 10709 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:14:37.869974 10709 webserver.cc:460] Webserver started at http://127.0.98.1:37857/ using document root <none> and password file <none>
I20260924 04:14:37.870473 10709 fs_manager.cc:359] Metadata directory not provided
I20260924 04:14:37.870527 10709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:14:37.870690 10709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:14:37.872694 10709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/ts-0/data/instance:
uuid: "ee8ab3f01a8e4aa592671c1842e70539"
format_stamp: "Formatted at 2026-09-24 04:14:37 on dist-test-slave-p401"
server_key: "a3b5c07c22f6c36b7f55b03654fb9ba6"
server_key_iv: "3a04d8afd44db594bf9c30d7e8ce2a5e"
server_key_version: "encryptionkey@0"
I20260924 04:14:37.873219 10709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ee8ab3f01a8e4aa592671c1842e70539"
format_stamp: "Formatted at 2026-09-24 04:14:37 on dist-test-slave-p401"
server_key: "a3b5c07c22f6c36b7f55b03654fb9ba6"
server_key_iv: "3a04d8afd44db594bf9c30d7e8ce2a5e"
server_key_version: "encryptionkey@0"
I20260924 04:14:37.876972 10709 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260924 04:14:37.879396 10724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:37.880694 10709 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 04:14:37.880793 10709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/ts-0/wal
uuid: "ee8ab3f01a8e4aa592671c1842e70539"
format_stamp: "Formatted at 2026-09-24 04:14:37 on dist-test-slave-p401"
server_key: "a3b5c07c22f6c36b7f55b03654fb9ba6"
server_key_iv: "3a04d8afd44db594bf9c30d7e8ce2a5e"
server_key_version: "encryptionkey@0"
I20260924 04:14:37.880893 10709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:14:37.890894 10709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:14:37.893868 10709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:14:37.894027 10709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:14:37.894557 10709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:14:37.895380 10709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:14:37.895427 10709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:37.895467 10709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:14:37.895495 10709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:37.903367 10709 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41183
I20260924 04:14:37.903388 10837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41183 every 8 connection(s)
I20260924 04:14:37.904240 10709 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:14:37.906857 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 10709
I20260924 04:14:37.906970 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:14:37.907301 392 external_mini_cluster.cc:1428] Setting key 899fea5608dce941557f9a1c7ed1b18c
Sep 24 04:14:37 dist-test-slave-p401 krb5kdc[10602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223277, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:14:37.918853 10686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:37.905965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56901 (local address 127.0.98.62:44493)
0924 04:14:37.906103 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:37.906107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:37.906987 (+ 880us) server_negotiation.cc:407] Connection header received
0924 04:14:37.908366 (+ 1379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:37.908369 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:37.908421 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:37.908494 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:37.909976 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:37.910501 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:37.912159 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:37.912422 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:37.912523 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:37.915268 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:37.915273 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:37.915276 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:37.915305 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:37.916770 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:37.917237 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:37.917240 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:37.917241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:37.917314 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:37.917650 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:37.917653 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:37.917655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:37.917876 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:37.917952 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:37.918560 (+ 608us) server_negotiation.cc:299] Negotiation successful
0924 04:14:37.918722 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:14:37.919904 10840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:37.906374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44493 (local address 127.0.98.1:56901)
0924 04:14:37.906804 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:37.906846 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:37.908155 (+ 1309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:37.908687 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:37.908697 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:37.909183 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:37.909825 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:37.909843 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:37.910626 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:37.910630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:37.912022 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:37.912033 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:37.912581 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:37.912586 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:37.912803 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:37.913430 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:37.913450 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:37.915055 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:14:37.916871 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:37.916876 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:37.916888 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:37.917150 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:37.917416 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:37.917419 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:37.917420 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:37.917559 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:37.917992 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:14:37.917997 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:14:37.918264 (+ 267us) client_negotiation.cc:769] Sending connection context
0924 04:14:37.918463 (+ 199us) client_negotiation.cc:240] Negotiation successful
0924 04:14:37.918786 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":114,"threads_started":1}
I20260924 04:14:37.921487 10838 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44493
I20260924 04:14:37.921734 10838 heartbeater.cc:464] Registering TS with master...
I20260924 04:14:37.922227 10838 heartbeater.cc:511] Master 127.0.98.62:44493 requested a full tablet report, sending...
I20260924 04:14:37.924016 10647 ts_manager.cc:194] Registered new tserver with Master: ee8ab3f01a8e4aa592671c1842e70539 (127.0.98.1:41183)
I20260924 04:14:37.925700 10647 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56901
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:14:37.934984 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44493
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:14:38.043180 10843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:14:38.043483 10843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:14:38.043526 10843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:14:38.047600 10843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:14:38.047673 10843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:14:38.047737 10843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:14:38.053119 10843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:14:38.054183 10843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:14:38.055197 10843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:14:38.064464 10843 server_base.cc:1034] running on GCE node
W20260924 04:14:38.064420 10848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:38.064424 10851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:38.064610 10849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:38.065028 10843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:14:38.065649 10843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:14:38.066805 10843 hybrid_clock.cc:648] HybridClock initialized: now 1790223278066784 us; error 37 us; skew 500 ppm
Sep 24 04:14:38 dist-test-slave-p401 krb5kdc[10602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223278, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:14:38.069465 10843 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:14:38.070315 10843 webserver.cc:460] Webserver started at http://127.0.98.2:46709/ using document root <none> and password file <none>
I20260924 04:14:38.070837 10843 fs_manager.cc:359] Metadata directory not provided
I20260924 04:14:38.070899 10843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:14:38.071079 10843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:14:38.073102 10843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/ts-1/data/instance:
uuid: "f59a666e98ee4bdf87bc3dad3cb392fa"
format_stamp: "Formatted at 2026-09-24 04:14:38 on dist-test-slave-p401"
server_key: "54978ca86e70f4720c7447afd371e6ec"
server_key_iv: "eb9c7848327e8d9797f577747f0ec5d5"
server_key_version: "encryptionkey@0"
I20260924 04:14:38.073657 10843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f59a666e98ee4bdf87bc3dad3cb392fa"
format_stamp: "Formatted at 2026-09-24 04:14:38 on dist-test-slave-p401"
server_key: "54978ca86e70f4720c7447afd371e6ec"
server_key_iv: "eb9c7848327e8d9797f577747f0ec5d5"
server_key_version: "encryptionkey@0"
I20260924 04:14:38.077464 10843 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260924 04:14:38.079917 10858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:38.081243 10843 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 04:14:38.081350 10843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/ts-1/wal
uuid: "f59a666e98ee4bdf87bc3dad3cb392fa"
format_stamp: "Formatted at 2026-09-24 04:14:38 on dist-test-slave-p401"
server_key: "54978ca86e70f4720c7447afd371e6ec"
server_key_iv: "eb9c7848327e8d9797f577747f0ec5d5"
server_key_version: "encryptionkey@0"
I20260924 04:14:38.081453 10843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:14:38.092769 10843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:14:38.095881 10843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:14:38.096053 10843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:14:38.096633 10843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:14:38.097451 10843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:14:38.097496 10843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:38.097548 10843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:14:38.097576 10843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:38.105747 10843 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34683
I20260924 04:14:38.105790 10971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34683 every 8 connection(s)
I20260924 04:14:38.106670 10843 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:14:38.110807 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 10843
I20260924 04:14:38.110934 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:14:38.111284 392 external_mini_cluster.cc:1428] Setting key 7ebda682445ade58265e6d85f95bccc6
Sep 24 04:14:38 dist-test-slave-p401 krb5kdc[10602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223278, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:14:38.121572 10686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:38.108372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44401 (local address 127.0.98.62:44493)
0924 04:14:38.108533 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:38.108537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:38.109372 (+ 835us) server_negotiation.cc:407] Connection header received
0924 04:14:38.110629 (+ 1257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:38.110631 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:38.110682 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:38.110744 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:38.112233 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:38.112886 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:38.114540 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:38.114855 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:38.114956 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:38.117688 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:38.117693 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:38.117696 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:38.117730 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:38.119440 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:38.119947 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:38.119951 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:38.119953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:38.120062 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:38.120387 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:38.120391 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:38.120393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:38.120653 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:38.120770 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:38.121240 (+ 470us) server_negotiation.cc:299] Negotiation successful
0924 04:14:38.121378 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 04:14:38.122325 10974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:38.108772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44493 (local address 127.0.98.2:44401)
0924 04:14:38.109220 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:38.109265 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:38.110411 (+ 1146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:38.110963 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:38.110972 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:38.111527 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:38.112100 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:38.112115 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:38.113033 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:38.113037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:38.114411 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:38.114424 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:38.115010 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:38.115013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:38.115181 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:38.115876 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:38.115893 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:38.117506 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:14:38.119560 (+ 2054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:38.119565 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:38.119580 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:38.119846 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:38.120169 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:38.120172 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:38.120173 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:38.120303 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:38.120752 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:14:38.120757 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:14:38.121001 (+ 244us) client_negotiation.cc:769] Sending connection context
0924 04:14:38.121155 (+ 154us) client_negotiation.cc:240] Negotiation successful
0924 04:14:38.121437 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":121,"threads_started":1}
I20260924 04:14:38.123579 10972 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44493
I20260924 04:14:38.123831 10972 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 04:14:38.124411 10972 heartbeater.cc:511] Master 127.0.98.62:44493 requested a full tablet report, sending...
I20260924 04:14:38.125633 10647 ts_manager.cc:194] Registered new tserver with Master: f59a666e98ee4bdf87bc3dad3cb392fa (127.0.98.2:34683)
I20260924 04:14:38.126272 10647 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44401
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:14:38.138132 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44493
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:14:38.248453 10977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:14:38.248739 10977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:14:38.248780 10977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:14:38.252779 10977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:14:38.252853 10977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:14:38.252933 10977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:14:38.258293 10977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:14:38.259392 10977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:14:38.260411 10977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:14:38.270337 10982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:38.270488 10977 server_base.cc:1034] running on GCE node
W20260924 04:14:38.270304 10983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:14:38.270318 10985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:14:38.271091 10977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:14:38.271763 10977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:14:38.272933 10977 hybrid_clock.cc:648] HybridClock initialized: now 1790223278272896 us; error 50 us; skew 500 ppm
Sep 24 04:14:38 dist-test-slave-p401 krb5kdc[10602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223278, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:14:38.275579 10977 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:14:38.276440 10977 webserver.cc:460] Webserver started at http://127.0.98.3:46427/ using document root <none> and password file <none>
I20260924 04:14:38.276978 10977 fs_manager.cc:359] Metadata directory not provided
I20260924 04:14:38.277038 10977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:14:38.277221 10977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:14:38.279219 10977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/ts-2/data/instance:
uuid: "12cf3bdd3f224b8f916c90a447b72fb4"
format_stamp: "Formatted at 2026-09-24 04:14:38 on dist-test-slave-p401"
server_key: "b202c718147eb3cd9df2a47a46758d9d"
server_key_iv: "60fe5419d3abff34284cb2d6ebf5c7a6"
server_key_version: "encryptionkey@0"
I20260924 04:14:38.279814 10977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/ts-2/wal/instance:
uuid: "12cf3bdd3f224b8f916c90a447b72fb4"
format_stamp: "Formatted at 2026-09-24 04:14:38 on dist-test-slave-p401"
server_key: "b202c718147eb3cd9df2a47a46758d9d"
server_key_iv: "60fe5419d3abff34284cb2d6ebf5c7a6"
server_key_version: "encryptionkey@0"
I20260924 04:14:38.283748 10977 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260924 04:14:38.286336 10992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:38.287746 10977 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:14:38.287851 10977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/ts-2/wal
uuid: "12cf3bdd3f224b8f916c90a447b72fb4"
format_stamp: "Formatted at 2026-09-24 04:14:38 on dist-test-slave-p401"
server_key: "b202c718147eb3cd9df2a47a46758d9d"
server_key_iv: "60fe5419d3abff34284cb2d6ebf5c7a6"
server_key_version: "encryptionkey@0"
I20260924 04:14:38.287942 10977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:14:38.303602 10977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:14:38.306389 10977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:14:38.306547 10977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:14:38.307090 10977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:14:38.308009 10977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:14:38.308058 10977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:38.308096 10977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:14:38.308116 10977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:14:38.316531 10977 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36765
I20260924 04:14:38.316552 11105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36765 every 8 connection(s)
I20260924 04:14:38.317504 10977 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:14:38.324120 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 10977
I20260924 04:14:38.324252 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:14:38.324572 392 external_mini_cluster.cc:1428] Setting key 9828ed323e5499e7b7d88e506c5fa7b7
Sep 24 04:14:38 dist-test-slave-p401 krb5kdc[10602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223278, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:14:38.331329 10686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:38.319339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44503 (local address 127.0.98.62:44493)
0924 04:14:38.319460 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:38.319464 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:38.320374 (+ 910us) server_negotiation.cc:407] Connection header received
0924 04:14:38.321700 (+ 1326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:38.321704 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:38.321763 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:38.321844 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:38.323046 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:38.323618 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:38.325135 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:38.325420 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:38.325546 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:38.328139 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:38.328142 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:38.328144 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:38.328169 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:38.329439 (+ 1270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:38.329962 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:38.329965 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:38.329966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:38.330029 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:38.330324 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:38.330326 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:38.330327 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:38.330487 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:38.330555 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:38.331067 (+ 512us) server_negotiation.cc:299] Negotiation successful
0924 04:14:38.331162 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 04:14:38.332261 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:38.319768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44493 (local address 127.0.98.3:44503)
0924 04:14:38.320219 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:38.320264 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:38.321478 (+ 1214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:38.322028 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:38.322036 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:38.322407 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:14:38.322903 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:38.322916 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:38.323732 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:38.323735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:38.324994 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:38.325003 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:38.325579 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:38.325583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:38.325781 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:38.326374 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:14:38.326397 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:14:38.327979 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:14:38.329542 (+ 1563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:38.329548 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:38.329562 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:38.329881 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:38.330096 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:14:38.330099 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:14:38.330101 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:14:38.330252 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:14:38.330593 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:14:38.330599 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:14:38.330875 (+ 276us) client_negotiation.cc:769] Sending connection context
0924 04:14:38.331042 (+ 167us) client_negotiation.cc:240] Negotiation successful
0924 04:14:38.331389 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":131,"threads_started":1}
I20260924 04:14:38.333410 11106 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44493
I20260924 04:14:38.333670 11106 heartbeater.cc:464] Registering TS with master...
I20260924 04:14:38.334157 11106 heartbeater.cc:511] Master 127.0.98.62:44493 requested a full tablet report, sending...
I20260924 04:14:38.335187 10647 ts_manager.cc:194] Registered new tserver with Master: 12cf3bdd3f224b8f916c90a447b72fb4 (127.0.98.3:36765)
I20260924 04:14:38.335760 10647 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44503
I20260924 04:14:38.338152 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:14:38.346572 10686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:38.339932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60300 (local address 127.0.98.62:44493)
0924 04:14:38.340037 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:38.340040 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:38.340180 (+ 140us) server_negotiation.cc:407] Connection header received
0924 04:14:38.340285 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:38.340287 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:38.340330 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:38.340391 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:14:38.341029 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:38.341523 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:38.342310 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:38.342507 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:38.342572 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:38.344001 (+ 1429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:14:38.344004 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:14:38.344006 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:14:38.344028 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:14:38.345143 (+ 1115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:38.345537 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:38.345540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:38.345541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:38.345607 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:14:38.345869 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:14:38.345872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:14:38.345873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:14:38.346069 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:14:38.346164 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:38.346360 (+ 196us) server_negotiation.cc:299] Negotiation successful
0924 04:14:38.346451 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 04:14:38.349206 10647 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60300:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:14:38.352218 10647 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:14:38.366191 11120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:38.360737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36765 (local address 127.0.0.1:34916)
0924 04:14:38.361568 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:38.361580 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:38.361653 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:38.362387 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:38.362390 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:38.362405 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:14:38.362485 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:38.362491 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:38.363484 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:38.363487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:38.364728 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:38.364736 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:38.365652 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:38.365655 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:38.365767 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:38.365810 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 04:14:38.365911 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 04:14:38.366035 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":775,"mutex_wait_us":69,"thread_start_us":82,"threads_started":1}
I20260924 04:14:38.366353 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:38.360292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56688 (local address 127.0.98.1:41183)
0924 04:14:38.360790 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:38.360814 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:38.361158 (+ 344us) server_negotiation.cc:407] Connection header received
0924 04:14:38.361238 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:38.361243 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:38.361451 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:38.361598 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:14:38.362138 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:38.363344 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:38.365174 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:38.365777 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:38.365883 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:38.366017 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:38.366112 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 04:14:38.366207 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":124,"threads_started":1}
I20260924 04:14:38.366736 11121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:38.360723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37720 (local address 127.0.98.2:34683)
0924 04:14:38.361144 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:38.361155 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:38.361172 (+ 17us) server_negotiation.cc:407] Connection header received
0924 04:14:38.361238 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:38.361243 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:38.361451 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:38.361598 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:14:38.362107 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:38.363358 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:38.365152 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:38.365902 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:38.365992 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:38.366118 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:38.366565 (+ 447us) server_negotiation.cc:299] Negotiation successful
0924 04:14:38.366639 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":141,"threads_started":1}
I20260924 04:14:38.366794 11122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:38.361099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34916 (local address 127.0.98.3:36765)
0924 04:14:38.361992 (+ 893us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:38.362002 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:38.362024 (+ 22us) server_negotiation.cc:407] Connection header received
0924 04:14:38.362093 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:38.362098 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:38.362260 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:38.362390 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:14:38.362583 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:38.363358 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:38.364843 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:38.365531 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:38.365636 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:38.365783 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:38.366588 (+ 805us) server_negotiation.cc:299] Negotiation successful
0924 04:14:38.366675 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":799,"thread_start_us":129,"threads_started":1}
I20260924 04:14:38.366360 11117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:38.360142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41183 (local address 127.0.0.1:56688)
0924 04:14:38.360946 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:38.360958 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:38.361054 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:38.361778 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:38.361784 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:38.361816 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:14:38.361995 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:38.362003 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:38.363496 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:38.363499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:38.365037 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:38.365044 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:38.365895 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:38.365899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:38.365982 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:38.366004 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 04:14:38.366069 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 04:14:38.366142 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":746,"mutex_wait_us":192,"thread_start_us":111,"threads_started":1}
I20260924 04:14:38.366495 11118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:38.360009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34683 (local address 127.0.0.1:37720)
0924 04:14:38.360511 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:38.360559 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:38.360840 (+ 281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:38.361778 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:38.361784 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:38.361807 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:14:38.361996 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:38.362003 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:38.363496 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:38.363499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:38.365034 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:38.365044 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:38.366170 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:38.366173 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:38.366258 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:38.366279 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 04:14:38.366341 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 04:14:38.366402 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":253,"thread_start_us":89,"threads_started":1}
I20260924 04:14:38.369688 10906 tablet_service.cc:1467] Processing CreateTablet for tablet 148b4915edab431d89e32b97b011d217 (DEFAULT_TABLE table=test-table [id=9dc902143c754a3c8ee76f5b1551d9d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:14:38.369822 11040 tablet_service.cc:1467] Processing CreateTablet for tablet 148b4915edab431d89e32b97b011d217 (DEFAULT_TABLE table=test-table [id=9dc902143c754a3c8ee76f5b1551d9d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:14:38.370177 10772 tablet_service.cc:1467] Processing CreateTablet for tablet 148b4915edab431d89e32b97b011d217 (DEFAULT_TABLE table=test-table [id=9dc902143c754a3c8ee76f5b1551d9d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:14:38.370671 10906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 148b4915edab431d89e32b97b011d217. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:14:38.370838 11040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 148b4915edab431d89e32b97b011d217. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:14:38.371057 10772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 148b4915edab431d89e32b97b011d217. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:14:38.377125 11123 tablet_bootstrap.cc:492] T 148b4915edab431d89e32b97b011d217 P f59a666e98ee4bdf87bc3dad3cb392fa: Bootstrap starting.
I20260924 04:14:38.379004 11124 tablet_bootstrap.cc:492] T 148b4915edab431d89e32b97b011d217 P 12cf3bdd3f224b8f916c90a447b72fb4: Bootstrap starting.
I20260924 04:14:38.379647 11125 tablet_bootstrap.cc:492] T 148b4915edab431d89e32b97b011d217 P ee8ab3f01a8e4aa592671c1842e70539: Bootstrap starting.
I20260924 04:14:38.379689 11123 tablet_bootstrap.cc:654] T 148b4915edab431d89e32b97b011d217 P f59a666e98ee4bdf87bc3dad3cb392fa: Neither blocks nor log segments found. Creating new log.
I20260924 04:14:38.380422 11123 log.cc:824] T 148b4915edab431d89e32b97b011d217 P f59a666e98ee4bdf87bc3dad3cb392fa: Log is configured to *not* fsync() on all Append() calls
I20260924 04:14:38.381345 11124 tablet_bootstrap.cc:654] T 148b4915edab431d89e32b97b011d217 P 12cf3bdd3f224b8f916c90a447b72fb4: Neither blocks nor log segments found. Creating new log.
I20260924 04:14:38.381901 11125 tablet_bootstrap.cc:654] T 148b4915edab431d89e32b97b011d217 P ee8ab3f01a8e4aa592671c1842e70539: Neither blocks nor log segments found. Creating new log.
I20260924 04:14:38.382470 11123 tablet_bootstrap.cc:492] T 148b4915edab431d89e32b97b011d217 P f59a666e98ee4bdf87bc3dad3cb392fa: No bootstrap required, opened a new log
I20260924 04:14:38.382664 11123 ts_tablet_manager.cc:1397] T 148b4915edab431d89e32b97b011d217 P f59a666e98ee4bdf87bc3dad3cb392fa: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260924 04:14:38.382862 11124 log.cc:824] T 148b4915edab431d89e32b97b011d217 P 12cf3bdd3f224b8f916c90a447b72fb4: Log is configured to *not* fsync() on all Append() calls
I20260924 04:14:38.383052 11125 log.cc:824] T 148b4915edab431d89e32b97b011d217 P ee8ab3f01a8e4aa592671c1842e70539: Log is configured to *not* fsync() on all Append() calls
I20260924 04:14:38.385056 11124 tablet_bootstrap.cc:492] T 148b4915edab431d89e32b97b011d217 P 12cf3bdd3f224b8f916c90a447b72fb4: No bootstrap required, opened a new log
I20260924 04:14:38.385056 11125 tablet_bootstrap.cc:492] T 148b4915edab431d89e32b97b011d217 P ee8ab3f01a8e4aa592671c1842e70539: No bootstrap required, opened a new log
I20260924 04:14:38.385277 11124 ts_tablet_manager.cc:1397] T 148b4915edab431d89e32b97b011d217 P 12cf3bdd3f224b8f916c90a447b72fb4: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20260924 04:14:38.385277 11125 ts_tablet_manager.cc:1397] T 148b4915edab431d89e32b97b011d217 P ee8ab3f01a8e4aa592671c1842e70539: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:14:38.386272 11123 raft_consensus.cc:348] T 148b4915edab431d89e32b97b011d217 P f59a666e98ee4bdf87bc3dad3cb392fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f59a666e98ee4bdf87bc3dad3cb392fa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34683 } } peers { permanent_uuid: "ee8ab3f01a8e4aa592671c1842e70539" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41183 } } peers { permanent_uuid: "12cf3bdd3f224b8f916c90a447b72fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36765 } }
I20260924 04:14:38.386572 11123 raft_consensus.cc:374] T 148b4915edab431d89e32b97b011d217 P f59a666e98ee4bdf87bc3dad3cb392fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:14:38.386642 11123 raft_consensus.cc:729] T 148b4915edab431d89e32b97b011d217 P f59a666e98ee4bdf87bc3dad3cb392fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f59a666e98ee4bdf87bc3dad3cb392fa, State: Initialized, Role: FOLLOWER
I20260924 04:14:38.387110 11123 consensus_queue.cc:260] T 148b4915edab431d89e32b97b011d217 P f59a666e98ee4bdf87bc3dad3cb392fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f59a666e98ee4bdf87bc3dad3cb392fa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34683 } } peers { permanent_uuid: "ee8ab3f01a8e4aa592671c1842e70539" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41183 } } peers { permanent_uuid: "12cf3bdd3f224b8f916c90a447b72fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36765 } }
I20260924 04:14:38.388247 11123 ts_tablet_manager.cc:1428] T 148b4915edab431d89e32b97b011d217 P f59a666e98ee4bdf87bc3dad3cb392fa: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 04:14:38.388681 10972 heartbeater.cc:503] Master 127.0.98.62:44493 was elected leader, sending a full tablet report...
I20260924 04:14:38.389297 11124 raft_consensus.cc:348] T 148b4915edab431d89e32b97b011d217 P 12cf3bdd3f224b8f916c90a447b72fb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f59a666e98ee4bdf87bc3dad3cb392fa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34683 } } peers { permanent_uuid: "ee8ab3f01a8e4aa592671c1842e70539" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41183 } } peers { permanent_uuid: "12cf3bdd3f224b8f916c90a447b72fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36765 } }
I20260924 04:14:38.389297 11125 raft_consensus.cc:348] T 148b4915edab431d89e32b97b011d217 P ee8ab3f01a8e4aa592671c1842e70539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f59a666e98ee4bdf87bc3dad3cb392fa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34683 } } peers { permanent_uuid: "ee8ab3f01a8e4aa592671c1842e70539" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41183 } } peers { permanent_uuid: "12cf3bdd3f224b8f916c90a447b72fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36765 } }
I20260924 04:14:38.389571 11124 raft_consensus.cc:374] T 148b4915edab431d89e32b97b011d217 P 12cf3bdd3f224b8f916c90a447b72fb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:14:38.389571 11125 raft_consensus.cc:374] T 148b4915edab431d89e32b97b011d217 P ee8ab3f01a8e4aa592671c1842e70539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:14:38.389621 11124 raft_consensus.cc:729] T 148b4915edab431d89e32b97b011d217 P 12cf3bdd3f224b8f916c90a447b72fb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12cf3bdd3f224b8f916c90a447b72fb4, State: Initialized, Role: FOLLOWER
I20260924 04:14:38.389626 11125 raft_consensus.cc:729] T 148b4915edab431d89e32b97b011d217 P ee8ab3f01a8e4aa592671c1842e70539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee8ab3f01a8e4aa592671c1842e70539, State: Initialized, Role: FOLLOWER
I20260924 04:14:38.390097 11125 consensus_queue.cc:260] T 148b4915edab431d89e32b97b011d217 P ee8ab3f01a8e4aa592671c1842e70539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f59a666e98ee4bdf87bc3dad3cb392fa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34683 } } peers { permanent_uuid: "ee8ab3f01a8e4aa592671c1842e70539" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41183 } } peers { permanent_uuid: "12cf3bdd3f224b8f916c90a447b72fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36765 } }
I20260924 04:14:38.390097 11124 consensus_queue.cc:260] T 148b4915edab431d89e32b97b011d217 P 12cf3bdd3f224b8f916c90a447b72fb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f59a666e98ee4bdf87bc3dad3cb392fa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34683 } } peers { permanent_uuid: "ee8ab3f01a8e4aa592671c1842e70539" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41183 } } peers { permanent_uuid: "12cf3bdd3f224b8f916c90a447b72fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36765 } }
I20260924 04:14:38.390986 10838 heartbeater.cc:503] Master 127.0.98.62:44493 was elected leader, sending a full tablet report...
I20260924 04:14:38.391178 11125 ts_tablet_manager.cc:1428] T 148b4915edab431d89e32b97b011d217 P ee8ab3f01a8e4aa592671c1842e70539: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 04:14:38.391178 11124 ts_tablet_manager.cc:1428] T 148b4915edab431d89e32b97b011d217 P 12cf3bdd3f224b8f916c90a447b72fb4: Time spent starting tablet: real 0.006s user 0.004s sys 0.002s
I20260924 04:14:38.392150 11106 heartbeater.cc:503] Master 127.0.98.62:44493 was elected leader, sending a full tablet report...
W20260924 04:14:38.405369 10839 tablet.cc:2369] T 148b4915edab431d89e32b97b011d217 P ee8ab3f01a8e4aa592671c1842e70539: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:14:38.468163 11131 raft_consensus.cc:482] T 148b4915edab431d89e32b97b011d217 P 12cf3bdd3f224b8f916c90a447b72fb4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:14:38.468387 11131 raft_consensus.cc:504] T 148b4915edab431d89e32b97b011d217 P 12cf3bdd3f224b8f916c90a447b72fb4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f59a666e98ee4bdf87bc3dad3cb392fa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34683 } } peers { permanent_uuid: "ee8ab3f01a8e4aa592671c1842e70539" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41183 } } peers { permanent_uuid: "12cf3bdd3f224b8f916c90a447b72fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36765 } }
I20260924 04:14:38.469491 11131 leader_election.cc:288] T 148b4915edab431d89e32b97b011d217 P 12cf3bdd3f224b8f916c90a447b72fb4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f59a666e98ee4bdf87bc3dad3cb392fa (127.0.98.2:34683), ee8ab3f01a8e4aa592671c1842e70539 (127.0.98.1:41183)
I20260924 04:14:38.474455 11121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:38.469826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47361 (local address 127.0.98.2:34683)
0924 04:14:38.469953 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:38.469958 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:38.469974 (+ 16us) server_negotiation.cc:407] Connection header received
0924 04:14:38.470143 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:38.470146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:38.470193 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:38.470279 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:14:38.470517 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:38.471355 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:38.473130 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:38.473752 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:38.473867 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:38.473932 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:38.474221 (+ 289us) server_negotiation.cc:299] Negotiation successful
0924 04:14:38.474312 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:14:38.474455 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:38.469972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48649 (local address 127.0.98.1:41183)
0924 04:14:38.470089 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:38.470093 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:38.470213 (+ 120us) server_negotiation.cc:407] Connection header received
0924 04:14:38.470309 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:38.470312 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:38.470357 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:38.470421 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:14:38.470638 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:38.471384 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:38.473130 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:38.473752 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:38.473863 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:38.473932 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:38.474221 (+ 289us) server_negotiation.cc:299] Negotiation successful
0924 04:14:38.474312 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:14:38.474468 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:38.469734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34683 (local address 127.0.98.3:47361)
0924 04:14:38.469904 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:38.469920 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:38.470017 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:38.470298 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:38.470300 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:38.470312 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:14:38.470412 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:38.470418 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:38.471496 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:38.471500 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:38.472990 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:38.473003 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:38.473871 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:38.473874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:38.474022 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:38.474087 (+ 65us) client_negotiation.cc:769] Sending connection context
0924 04:14:38.474164 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 04:14:38.474289 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":151}
I20260924 04:14:38.474476 11132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:38.469758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41183 (local address 127.0.98.3:48649)
0924 04:14:38.470122 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:38.470132 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 04:14:38.470216 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:14:38.470442 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:14:38.470444 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:14:38.470457 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:14:38.470550 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:38.470554 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:38.471505 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:38.471508 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:38.472990 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:14:38.473003 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:14:38.473871 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:14:38.473875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:14:38.474003 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:38.474085 (+ 82us) client_negotiation.cc:769] Sending connection context
0924 04:14:38.474171 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 04:14:38.474288 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":189,"thread_start_us":125,"threads_started":1}
I20260924 04:14:38.475265 10926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "148b4915edab431d89e32b97b011d217" candidate_uuid: "12cf3bdd3f224b8f916c90a447b72fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f59a666e98ee4bdf87bc3dad3cb392fa" is_pre_election: true
I20260924 04:14:38.475431 10792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "148b4915edab431d89e32b97b011d217" candidate_uuid: "12cf3bdd3f224b8f916c90a447b72fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee8ab3f01a8e4aa592671c1842e70539" is_pre_election: true
I20260924 04:14:38.475605 10926 raft_consensus.cc:2454] T 148b4915edab431d89e32b97b011d217 P f59a666e98ee4bdf87bc3dad3cb392fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12cf3bdd3f224b8f916c90a447b72fb4 in term 0.
I20260924 04:14:38.475781 10792 raft_consensus.cc:2454] T 148b4915edab431d89e32b97b011d217 P ee8ab3f01a8e4aa592671c1842e70539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12cf3bdd3f224b8f916c90a447b72fb4 in term 0.
I20260924 04:14:38.476085 10995 leader_election.cc:302] T 148b4915edab431d89e32b97b011d217 P 12cf3bdd3f224b8f916c90a447b72fb4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12cf3bdd3f224b8f916c90a447b72fb4, f59a666e98ee4bdf87bc3dad3cb392fa; no voters:
I20260924 04:14:38.476380 11131 raft_consensus.cc:2781] T 148b4915edab431d89e32b97b011d217 P 12cf3bdd3f224b8f916c90a447b72fb4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:14:38.476470 11131 raft_consensus.cc:482] T 148b4915edab431d89e32b97b011d217 P 12cf3bdd3f224b8f916c90a447b72fb4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:14:38.476508 11131 raft_consensus.cc:3037] T 148b4915edab431d89e32b97b011d217 P 12cf3bdd3f224b8f916c90a447b72fb4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:14:38.477648 11131 raft_consensus.cc:504] T 148b4915edab431d89e32b97b011d217 P 12cf3bdd3f224b8f916c90a447b72fb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f59a666e98ee4bdf87bc3dad3cb392fa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34683 } } peers { permanent_uuid: "ee8ab3f01a8e4aa592671c1842e70539" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41183 } } peers { permanent_uuid: "12cf3bdd3f224b8f916c90a447b72fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36765 } }
I20260924 04:14:38.478044 11131 leader_election.cc:288] T 148b4915edab431d89e32b97b011d217 P 12cf3bdd3f224b8f916c90a447b72fb4 [CANDIDATE]: Term 1 election: Requested vote from peers f59a666e98ee4bdf87bc3dad3cb392fa (127.0.98.2:34683), ee8ab3f01a8e4aa592671c1842e70539 (127.0.98.1:41183)
I20260924 04:14:38.478449 10792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "148b4915edab431d89e32b97b011d217" candidate_uuid: "12cf3bdd3f224b8f916c90a447b72fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee8ab3f01a8e4aa592671c1842e70539"
I20260924 04:14:38.478565 10926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "148b4915edab431d89e32b97b011d217" candidate_uuid: "12cf3bdd3f224b8f916c90a447b72fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f59a666e98ee4bdf87bc3dad3cb392fa"
I20260924 04:14:38.478574 10792 raft_consensus.cc:3037] T 148b4915edab431d89e32b97b011d217 P ee8ab3f01a8e4aa592671c1842e70539 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:14:38.478734 10926 raft_consensus.cc:3037] T 148b4915edab431d89e32b97b011d217 P f59a666e98ee4bdf87bc3dad3cb392fa [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:14:38.479810 10926 raft_consensus.cc:2454] T 148b4915edab431d89e32b97b011d217 P f59a666e98ee4bdf87bc3dad3cb392fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12cf3bdd3f224b8f916c90a447b72fb4 in term 1.
I20260924 04:14:38.479830 10792 raft_consensus.cc:2454] T 148b4915edab431d89e32b97b011d217 P ee8ab3f01a8e4aa592671c1842e70539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12cf3bdd3f224b8f916c90a447b72fb4 in term 1.
I20260924 04:14:38.480129 10996 leader_election.cc:302] T 148b4915edab431d89e32b97b011d217 P 12cf3bdd3f224b8f916c90a447b72fb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12cf3bdd3f224b8f916c90a447b72fb4, ee8ab3f01a8e4aa592671c1842e70539; no voters:
I20260924 04:14:38.480317 11131 raft_consensus.cc:2781] T 148b4915edab431d89e32b97b011d217 P 12cf3bdd3f224b8f916c90a447b72fb4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:14:38.480556 11131 raft_consensus.cc:686] T 148b4915edab431d89e32b97b011d217 P 12cf3bdd3f224b8f916c90a447b72fb4 [term 1 LEADER]: Becoming Leader. State: Replica: 12cf3bdd3f224b8f916c90a447b72fb4, State: Running, Role: LEADER
I20260924 04:14:38.480922 11131 consensus_queue.cc:237] T 148b4915edab431d89e32b97b011d217 P 12cf3bdd3f224b8f916c90a447b72fb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f59a666e98ee4bdf87bc3dad3cb392fa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34683 } } peers { permanent_uuid: "ee8ab3f01a8e4aa592671c1842e70539" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41183 } } peers { permanent_uuid: "12cf3bdd3f224b8f916c90a447b72fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36765 } }
I20260924 04:14:38.484211 10647 catalog_manager.cc:5495] T 148b4915edab431d89e32b97b011d217 P 12cf3bdd3f224b8f916c90a447b72fb4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12cf3bdd3f224b8f916c90a447b72fb4 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "12cf3bdd3f224b8f916c90a447b72fb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f59a666e98ee4bdf87bc3dad3cb392fa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee8ab3f01a8e4aa592671c1842e70539" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12cf3bdd3f224b8f916c90a447b72fb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36765 } health_report { overall_health: HEALTHY } } }
I20260924 04:14:38.518596 11122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:38.515095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34922 (local address 127.0.98.3:36765)
0924 04:14:38.515252 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 04:14:38.515257 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:14:38.515275 (+ 18us) server_negotiation.cc:407] Connection header received
0924 04:14:38.515353 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:14:38.515355 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:14:38.515414 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:14:38.515507 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:14:38.515787 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:38.516284 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:38.517172 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:14:38.517378 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:14:38.517444 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:14:38.517685 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:14:38.518117 (+ 432us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:14:38.518207 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 04:14:38.518396 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 04:14:38.518460 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 04:14:38.524613 10792 raft_consensus.cc:1261] T 148b4915edab431d89e32b97b011d217 P ee8ab3f01a8e4aa592671c1842e70539 [term 1 FOLLOWER]: Refusing update from remote peer 12cf3bdd3f224b8f916c90a447b72fb4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:14:38.524613 10926 raft_consensus.cc:1261] T 148b4915edab431d89e32b97b011d217 P f59a666e98ee4bdf87bc3dad3cb392fa [term 1 FOLLOWER]: Refusing update from remote peer 12cf3bdd3f224b8f916c90a447b72fb4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:14:38.525346 11133 consensus_queue.cc:1035] T 148b4915edab431d89e32b97b011d217 P 12cf3bdd3f224b8f916c90a447b72fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee8ab3f01a8e4aa592671c1842e70539" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:14:38.525517 11131 consensus_queue.cc:1035] T 148b4915edab431d89e32b97b011d217 P 12cf3bdd3f224b8f916c90a447b72fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f59a666e98ee4bdf87bc3dad3cb392fa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:14:38.532212 11136 mvcc.cc:205] Tried to move back new op lower bound from 7332754548829523968 to 7332754548660727808. Current Snapshot: MvccSnapshot[applied={T|T < 7332754548829523968}]
I20260924 04:14:38.542073 10647 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60300:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:14:38.542291 10647 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60300:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:14:38.544909 10647 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2e35a25996be41b691945ea647935748: Sending DeleteTablet for 3 replicas of tablet 148b4915edab431d89e32b97b011d217
I20260924 04:14:38.545764 10906 tablet_service.cc:1514] Processing DeleteTablet for tablet 148b4915edab431d89e32b97b011d217 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:14:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37720
I20260924 04:14:38.545765 10772 tablet_service.cc:1514] Processing DeleteTablet for tablet 148b4915edab431d89e32b97b011d217 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:14:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56688
I20260924 04:14:38.546293 11040 tablet_service.cc:1514] Processing DeleteTablet for tablet 148b4915edab431d89e32b97b011d217 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:14:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34916
I20260924 04:14:38.546723 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 10709
I20260924 04:14:38.546584 11145 tablet_replica.cc:330] T 148b4915edab431d89e32b97b011d217 P f59a666e98ee4bdf87bc3dad3cb392fa: stopping tablet replica
I20260924 04:14:38.548684 11147 tablet_replica.cc:330] T 148b4915edab431d89e32b97b011d217 P 12cf3bdd3f224b8f916c90a447b72fb4: stopping tablet replica
I20260924 04:14:38.548698 11145 raft_consensus.cc:2229] T 148b4915edab431d89e32b97b011d217 P f59a666e98ee4bdf87bc3dad3cb392fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:14:38.548986 11147 raft_consensus.cc:2229] T 148b4915edab431d89e32b97b011d217 P 12cf3bdd3f224b8f916c90a447b72fb4 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:14:38.549052 11145 raft_consensus.cc:2258] T 148b4915edab431d89e32b97b011d217 P f59a666e98ee4bdf87bc3dad3cb392fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:14:38.549412 11147 raft_consensus.cc:2258] T 148b4915edab431d89e32b97b011d217 P 12cf3bdd3f224b8f916c90a447b72fb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:14:38.550815 11145 ts_tablet_manager.cc:1905] T 148b4915edab431d89e32b97b011d217 P f59a666e98ee4bdf87bc3dad3cb392fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:14:38.551287 11147 ts_tablet_manager.cc:1905] T 148b4915edab431d89e32b97b011d217 P 12cf3bdd3f224b8f916c90a447b72fb4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:14:38.553753 11145 ts_tablet_manager.cc:1918] T 148b4915edab431d89e32b97b011d217 P f59a666e98ee4bdf87bc3dad3cb392fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:14:38.553893 11147 ts_tablet_manager.cc:1918] T 148b4915edab431d89e32b97b011d217 P 12cf3bdd3f224b8f916c90a447b72fb4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:14:38.553939 11145 log.cc:1196] T 148b4915edab431d89e32b97b011d217 P f59a666e98ee4bdf87bc3dad3cb392fa: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/ts-1/wal/wals/148b4915edab431d89e32b97b011d217
I20260924 04:14:38.554032 11147 log.cc:1196] T 148b4915edab431d89e32b97b011d217 P 12cf3bdd3f224b8f916c90a447b72fb4: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790223216265354-392-0/minicluster-data/ts-2/wal/wals/148b4915edab431d89e32b97b011d217
I20260924 04:14:38.554363 11145 ts_tablet_manager.cc:1939] T 148b4915edab431d89e32b97b011d217 P f59a666e98ee4bdf87bc3dad3cb392fa: Deleting consensus metadata
I20260924 04:14:38.554378 11147 ts_tablet_manager.cc:1939] T 148b4915edab431d89e32b97b011d217 P 12cf3bdd3f224b8f916c90a447b72fb4: Deleting consensus metadata
W20260924 04:14:38.554554 10634 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41183 (error 108)
I20260924 04:14:38.555475 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 10843
I20260924 04:14:38.555625 10634 catalog_manager.cc:4841] TS 12cf3bdd3f224b8f916c90a447b72fb4 (127.0.98.3:36765): tablet 148b4915edab431d89e32b97b011d217 (table test-table [id=9dc902143c754a3c8ee76f5b1551d9d4]) successfully deleted
I20260924 04:14:38.555631 10633 catalog_manager.cc:4841] TS f59a666e98ee4bdf87bc3dad3cb392fa (127.0.98.2:34683): tablet 148b4915edab431d89e32b97b011d217 (table test-table [id=9dc902143c754a3c8ee76f5b1551d9d4]) successfully deleted
I20260924 04:14:38.561236 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 10977
I20260924 04:14:38.562812 11117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:14:38.562466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41183 (local address 127.0.0.1:56690)
0924 04:14:38.562657 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 04:14:38.562730 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
W20260924 04:14:38.562987 10634 catalog_manager.cc:4584] TS ee8ab3f01a8e4aa592671c1842e70539 (127.0.98.1:41183): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 148b4915edab431d89e32b97b011d217: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41183: connect: Connection refused (error 111)
I20260924 04:14:38.566701 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 10615
2026-09-24T04:14:38Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2212 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:14:38 dist-test-slave-p401 krb5kdc[11151](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:14:38 dist-test-slave-p401 krb5kdc[11151](info): set up 2 sockets
Sep 24 04:14:38 dist-test-slave-p401 krb5kdc[11151](info): commencing operation
krb5kdc: starting...
W20260924 04:14:39.532056 392 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/krb5kdc/test-user.keytab.
Sep 24 04:14:39 dist-test-slave-p401 krb5kdc[11151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:14:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:14:39Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260924 04:14:39.698818 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task6HV92s/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/postgres -l logfile start
2026-09-24 04:14:42.160 UTC [11181] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 04:14:42.160 UTC [11181] LOG: listening on IPv4 address "127.0.98.20", port 42691
2026-09-24 04:14:42.164 UTC [11181] LOG: listening on Unix socket "/tmp/.s.PGSQL.42691"
2026-09-24 04:14:42.191 UTC [11184] LOG: database system was shut down at 2026-09-24 04:14:40 UTC
2026-09-24 04:14:42.195 UTC [11181] LOG: database system is ready to accept connections
I20260924 04:14:43.067763 392 mini_postgres.cc:96] Postgres bound to 42691
2026-09-24 04:14:43.073 UTC [11192] FATAL: database "slave" does not exist
127.0.98.20:42691 - accepting connections
I20260924 04:14:43.074405 392 mini_ranger.cc:161] Starting Ranger...
I20260924 04:14:43.088193 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 04:14:43.543110 392 mini_ranger.cc:87] Created ranger Postgres database
I20260924 04:14:43.543200 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ranger-admin
2026-09-24 04:14:43,808 [I] DB FLAVOR :POSTGRES
2026-09-24 04:14:43,809 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 04:14:43,809 [I] Checking connection..
2026-09-24 04:14:43,809 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6HV92s/build/debug/bin/postgresql.jar:/tmp/dist-test-task6HV92s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42691/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 04:14:44,033 [I] Checking connection passed.
2026-09-24 04:14:44,033 [I] --------- Verifying version history table ---------
2026-09-24 04:14:44,033 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6HV92s/build/debug/bin/postgresql.jar:/tmp/dist-test-task6HV92s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42691/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 04:14:44,289 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 04:14:44,289 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6HV92s/build/debug/bin/postgresql.jar:/tmp/dist-test-task6HV92s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42691/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 04:14:44,537 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 04:14:44,538 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 04:14:44,538 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6HV92s/build/debug/bin/postgresql.jar:/tmp/dist-test-task6HV92s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42691/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6HV92s/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 04:14:44.777 UTC [11256] WARNING: there is no transaction in progress
2026-09-24 04:14:44,783 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 04:14:44,783 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6HV92s/build/debug/bin/postgresql.jar:/tmp/dist-test-task6HV92s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42691/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 04:14:45,008 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 04:14:45,008 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 04:14:45,008 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6HV92s/build/debug/bin/postgresql.jar:/tmp/dist-test-task6HV92s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42691/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 04:14:45,230 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6HV92s/build/debug/bin/postgresql.jar:/tmp/dist-test-task6HV92s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42691/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 04:14:45,453 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6HV92s/build/debug/bin/postgresql.jar:/tmp/dist-test-task6HV92s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42691/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-p401.c.gcp-upstream.internal','N') ;"
2026-09-24 04:14:45,667 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6HV92s/build/debug/bin/postgresql.jar:/tmp/dist-test-task6HV92s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42691/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 04:14:45,902 [I] Table x_portal_user does not exist in database ranger
2026-09-24 04:14:45,902 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6HV92s/build/debug/bin/postgresql.jar:/tmp/dist-test-task6HV92s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42691/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 04:14:46,135 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 04:14:46,135 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6HV92s/build/debug/bin/postgresql.jar:/tmp/dist-test-task6HV92s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42691/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 04:14:46,381 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6HV92s/build/debug/bin/postgresql.jar:/tmp/dist-test-task6HV92s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42691/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 04:14:46,631 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 04:14:46,631 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6HV92s/build/debug/bin/postgresql.jar:/tmp/dist-test-task6HV92s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42691/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6HV92s/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 04:14:47.550 UTC [11364] WARNING: there is no transaction in progress
2026-09-24 04:14:47.566 UTC [11364] WARNING: there is no transaction in progress
2026-09-24 04:14:47.582 UTC [11364] WARNING: there is no transaction in progress
2026-09-24 04:14:47.598 UTC [11364] WARNING: there is no transaction in progress
2026-09-24 04:14:47.613 UTC [11364] WARNING: there is no transaction in progress
2026-09-24 04:14:47.630 UTC [11364] WARNING: there is no transaction in progress
2026-09-24 04:14:47.694 UTC [11364] WARNING: there is no transaction in progress
2026-09-24 04:14:47.700 UTC [11364] WARNING: there is no transaction in progress
2026-09-24 04:14:47.714 UTC [11364] WARNING: there is no transaction in progress
2026-09-24 04:14:47.724 UTC [11364] WARNING: there is no transaction in progress
2026-09-24 04:14:47.734 UTC [11364] WARNING: there is no transaction in progress
2026-09-24 04:14:47.747 UTC [11364] WARNING: there is no transaction in progress
2026-09-24 04:14:47.757 UTC [11364] WARNING: there is no transaction in progress
2026-09-24 04:14:47.763 UTC [11364] WARNING: there is no transaction in progress
2026-09-24 04:14:47.770 UTC [11364] WARNING: there is no transaction in progress
2026-09-24 04:14:47.777 UTC [11364] WARNING: there is no transaction in progress
2026-09-24 04:14:48,387 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 04:14:48,388 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6HV92s/build/debug/bin/postgresql.jar:/tmp/dist-test-task6HV92s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42691/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 04:14:48,611 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 04:14:48,611 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6HV92s/build/debug/bin/postgresql.jar:/tmp/dist-test-task6HV92s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42691/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 04:14:48,832 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 04:14:48,832 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6HV92s/build/debug/bin/postgresql.jar:/tmp/dist-test-task6HV92s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42691/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 04:14:49,060 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 04:14:49,060 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6HV92s/build/debug/bin/postgresql.jar:/tmp/dist-test-task6HV92s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42691/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 04:14:49,308 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6HV92s/build/debug/bin/postgresql.jar:/tmp/dist-test-task6HV92s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42691/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 04:14:49,552 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6HV92s/build/debug/bin/postgresql.jar:/tmp/dist-test-task6HV92s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42691/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-p401.c.gcp-upstream.internal';"
2026-09-24 04:14:49,764 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 04:14:49,764 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6HV92s/build/debug/bin/postgresql.jar:/tmp/dist-test-task6HV92s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42691/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 04:14:50,014 [I] DB_PATCHES have already been applied
I20260924 04:14:50.020117 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ranger-admin:/tmp/dist-test-task6HV92s/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task6HV92s/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task6HV92s/build/debug/bin/postgresql.jar:/tmp/dist-test-task6HV92s/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 04:14:50.020188 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260924 04:14:50.023450 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:40899
Sep 24, 2026 4:14:50 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 4:14:50 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 4:14:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 4:14:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 4:14:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 4:14:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 4:14:50 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 04:14:51.046789 392 mini_ranger.cc:160] Time spent starting Ranger: real 7.972s user 0.000s sys 0.009s
Sep 24, 2026 4:15:09 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 4:15:09 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 24, 2026 4:15:09 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 4:15:09 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 4:15:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 4:15:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 4:15:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 4:15:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 4:15:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 4:15:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 4:15:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 4:15:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 4:15:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 4:15:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 4:15:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 4:15:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 4:15:09 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 4:15:10 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 04:15:11.404138 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 04:15:11 dist-test-slave-p401 krb5kdc[11151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223311, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 04:15:11.484186 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 04:15:11.493229 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 04:15:12.346776 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 04:15:12.346861 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ranger-kms
I20260924 04:15:12.346939 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:42691
2026-09-24 04:15:12,523 [I] DB FLAVOR :POSTGRES
2026-09-24 04:15:12,523 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 04:15:12,523 [I] Checking connection
2026-09-24 04:15:12,523 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6HV92s/build/debug/bin/postgresql.jar:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42691/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 04:15:12,748 [I] connection success
2026-09-24 04:15:12,748 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 04:15:12,748 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 04:15:12,748 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6HV92s/build/debug/bin/postgresql.jar:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42691/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 04:15:12,970 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 04:15:12,971 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 04:15:12,971 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 04:15:12,971 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6HV92s/build/debug/bin/postgresql.jar:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42691/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 04:15:13,213 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 04:15:13.652884 392 mini_ranger_kms.cc:310] Created kms service
I20260924 04:15:13.858207 392 mini_ranger_kms.cc:326] Created kudu user
I20260924 04:15:13.900048 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 04:15:13 dist-test-slave-p401 krb5kdc[11151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223311, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:15:14.262396 392 mini_ranger_kms.cc:384] Added ranger policy
I20260924 04:15:14.262578 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ranger-kms:/tmp/dist-test-task6HV92s/build/debug/bin/postgresql.jar:/tmp/dist-test-task6HV92s/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task6HV92s/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 04:15:14.262627 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260924 04:15:14.265988 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11544
I20260924 04:15:14.266103 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:49937
Sep 24, 2026 4:15:15 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 4:15:15 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 4:15:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 4:15:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 4:15:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 4:15:15 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 04:15:16.221711 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.738s user 0.017s sys 0.143s
I20260924 04:15:16.221817 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 04:15:16.221899 392 mini_ranger_kms.cc:401] 127.0.98.20:49937/kms/v1/keys
Sep 24 04:15:16 dist-test-slave-p401 krb5kdc[11151](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790223316, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 04:15:16 dist-test-slave-p401 krb5kdc[11151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223316, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 04:15:17 dist-test-slave-p401 krb5kdc[11151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223316, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 04:15:19 dist-test-slave-p401 krb5kdc[11151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:15:19.362205 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40463
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40463
--ranger_config_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:49937/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:15:19.498453 11595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:15:19.498724 11595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:15:19.498761 11595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:15:19.502775 11595 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:15:19.502846 11595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:15:19.502864 11595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:15:19.502878 11595 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:15:19.502892 11595 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:15:19.508455 11595 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:49937/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40463
--ranger_config_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:15:19.509615 11595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:15:19.510654 11595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:15:19.519405 11603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:15:19.519519 11601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:15:19.519419 11595 server_base.cc:1034] running on GCE node
W20260924 04:15:19.519412 11600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:15:19.520288 11595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:15:19.521488 11595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:15:19.522656 11595 hybrid_clock.cc:648] HybridClock initialized: now 1790223319522630 us; error 43 us; skew 500 ppm
Sep 24 04:15:19 dist-test-slave-p401 krb5kdc[11151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223319, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:15:19.530121 11595 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:15:19.531322 11595 webserver.cc:460] Webserver started at http://127.0.98.62:42571/ using document root <none> and password file <none>
I20260924 04:15:19.532016 11595 fs_manager.cc:359] Metadata directory not provided
I20260924 04:15:19.532109 11595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:15:19.532366 11595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 04:15:19 dist-test-slave-p401 krb5kdc[11151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223319, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:15:19.585135 11595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/master-0/data/instance:
uuid: "c59a33a32c8a4e08a8814e3da7a2fcb1"
format_stamp: "Formatted at 2026-09-24 04:15:19 on dist-test-slave-p401"
server_key: "869e24843288347ef5ba98f838d51751"
server_key_iv: "37fc12d53e1f0501290bb14785f9ce5f"
server_key_version: "kuduclusterkey@0"
I20260924 04:15:19.585914 11595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/master-0/wal/instance:
uuid: "c59a33a32c8a4e08a8814e3da7a2fcb1"
format_stamp: "Formatted at 2026-09-24 04:15:19 on dist-test-slave-p401"
server_key: "869e24843288347ef5ba98f838d51751"
server_key_iv: "37fc12d53e1f0501290bb14785f9ce5f"
server_key_version: "kuduclusterkey@0"
I20260924 04:15:19.591384 11595 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260924 04:15:19.609913 11610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:15:19.611580 11595 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 04:15:19.611735 11595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/master-0/wal
uuid: "c59a33a32c8a4e08a8814e3da7a2fcb1"
format_stamp: "Formatted at 2026-09-24 04:15:19 on dist-test-slave-p401"
server_key: "869e24843288347ef5ba98f838d51751"
server_key_iv: "37fc12d53e1f0501290bb14785f9ce5f"
server_key_version: "kuduclusterkey@0"
I20260924 04:15:19.611860 11595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:15:19.630074 11595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:15:19.632812 11595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:15:19.632977 11595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:15:19.640127 11595 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40463
I20260924 04:15:19.640194 11662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40463 every 8 connection(s)
I20260924 04:15:19.641037 11595 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:15:19.644326 11663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:15:19.650238 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 11595
I20260924 04:15:19.650355 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/master-0/wal/instance
I20260924 04:15:19.650596 392 external_mini_cluster.cc:1428] Setting key acb40eae18a21e54df90b2d212ff3d7b
I20260924 04:15:19.651755 11663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c59a33a32c8a4e08a8814e3da7a2fcb1: Bootstrap starting.
I20260924 04:15:19.654691 11663 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c59a33a32c8a4e08a8814e3da7a2fcb1: Neither blocks nor log segments found. Creating new log.
I20260924 04:15:19.655580 11663 log.cc:824] T 00000000000000000000000000000000 P c59a33a32c8a4e08a8814e3da7a2fcb1: Log is configured to *not* fsync() on all Append() calls
I20260924 04:15:19.657784 11663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c59a33a32c8a4e08a8814e3da7a2fcb1: No bootstrap required, opened a new log
Sep 24 04:15:19 dist-test-slave-p401 krb5kdc[11151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:15:19.662323 11663 raft_consensus.cc:348] T 00000000000000000000000000000000 P c59a33a32c8a4e08a8814e3da7a2fcb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c59a33a32c8a4e08a8814e3da7a2fcb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40463 } }
I20260924 04:15:19.662645 11663 raft_consensus.cc:374] T 00000000000000000000000000000000 P c59a33a32c8a4e08a8814e3da7a2fcb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:15:19.662719 11663 raft_consensus.cc:729] T 00000000000000000000000000000000 P c59a33a32c8a4e08a8814e3da7a2fcb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c59a33a32c8a4e08a8814e3da7a2fcb1, State: Initialized, Role: FOLLOWER
I20260924 04:15:19.663249 11663 consensus_queue.cc:260] T 00000000000000000000000000000000 P c59a33a32c8a4e08a8814e3da7a2fcb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c59a33a32c8a4e08a8814e3da7a2fcb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40463 } }
I20260924 04:15:19.663421 11663 raft_consensus.cc:388] T 00000000000000000000000000000000 P c59a33a32c8a4e08a8814e3da7a2fcb1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:15:19.663501 11663 raft_consensus.cc:482] T 00000000000000000000000000000000 P c59a33a32c8a4e08a8814e3da7a2fcb1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:15:19.663625 11663 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c59a33a32c8a4e08a8814e3da7a2fcb1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:15:19.663699 11666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:15:19.652011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33274 (local address 127.0.98.62:40463)
0924 04:15:19.652392 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0924 04:15:19.652407 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:15:19.652441 (+ 34us) server_negotiation.cc:407] Connection header received
0924 04:15:19.653281 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:15:19.653310 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:15:19.653643 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:15:19.653946 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:15:19.654712 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:15:19.655768 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:15:19.656738 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:15:19.657189 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:15:19.657329 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:15:19.659582 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:15:19.659586 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:15:19.659599 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:15:19.659626 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:15:19.661061 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:15:19.661453 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:15:19.661458 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:15:19.661462 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:15:19.661562 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:15:19.661810 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:15:19.661813 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:15:19.661815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:15:19.662185 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:15:19.662365 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0924 04:15:19.662617 (+ 252us) server_negotiation.cc:299] Negotiation successful
0924 04:15:19.662890 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":105,"threads_started":1}
I20260924 04:15:19.664733 11663 raft_consensus.cc:504] T 00000000000000000000000000000000 P c59a33a32c8a4e08a8814e3da7a2fcb1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c59a33a32c8a4e08a8814e3da7a2fcb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40463 } }
I20260924 04:15:19.665097 11663 leader_election.cc:302] T 00000000000000000000000000000000 P c59a33a32c8a4e08a8814e3da7a2fcb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c59a33a32c8a4e08a8814e3da7a2fcb1; no voters:
I20260924 04:15:19.665432 11663 leader_election.cc:288] T 00000000000000000000000000000000 P c59a33a32c8a4e08a8814e3da7a2fcb1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:15:19.665511 11668 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c59a33a32c8a4e08a8814e3da7a2fcb1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:15:19.666065 11668 raft_consensus.cc:686] T 00000000000000000000000000000000 P c59a33a32c8a4e08a8814e3da7a2fcb1 [term 1 LEADER]: Becoming Leader. State: Replica: c59a33a32c8a4e08a8814e3da7a2fcb1, State: Running, Role: LEADER
I20260924 04:15:19.666388 11668 consensus_queue.cc:237] T 00000000000000000000000000000000 P c59a33a32c8a4e08a8814e3da7a2fcb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c59a33a32c8a4e08a8814e3da7a2fcb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40463 } }
I20260924 04:15:19.666813 11663 sys_catalog.cc:564] T 00000000000000000000000000000000 P c59a33a32c8a4e08a8814e3da7a2fcb1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:15:19.667917 11670 sys_catalog.cc:455] T 00000000000000000000000000000000 P c59a33a32c8a4e08a8814e3da7a2fcb1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c59a33a32c8a4e08a8814e3da7a2fcb1. Latest consensus state: current_term: 1 leader_uuid: "c59a33a32c8a4e08a8814e3da7a2fcb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c59a33a32c8a4e08a8814e3da7a2fcb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40463 } } }
I20260924 04:15:19.668051 11670 sys_catalog.cc:458] T 00000000000000000000000000000000 P c59a33a32c8a4e08a8814e3da7a2fcb1 [sys.catalog]: This master's current role is: LEADER
I20260924 04:15:19.667918 11669 sys_catalog.cc:455] T 00000000000000000000000000000000 P c59a33a32c8a4e08a8814e3da7a2fcb1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c59a33a32c8a4e08a8814e3da7a2fcb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c59a33a32c8a4e08a8814e3da7a2fcb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40463 } } }
I20260924 04:15:19.668164 11669 sys_catalog.cc:458] T 00000000000000000000000000000000 P c59a33a32c8a4e08a8814e3da7a2fcb1 [sys.catalog]: This master's current role is: LEADER
I20260924 04:15:19.668413 11677 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:15:19.671423 11677 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:15:19.671845 11663 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 04:15:19.677567 11677 catalog_manager.cc:1347] Generated new cluster ID: 352dd955fbcb454e81441cfa82073665
I20260924 04:15:19.677639 11677 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:15:19.700219 11677 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:15:19.701728 11677 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:15:19.711274 11677 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c59a33a32c8a4e08a8814e3da7a2fcb1: Generated new TSK 0
I20260924 04:15:19.712289 11677 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:15:20.378934 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:49937/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40463
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:15:20.514514 11708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:15:20.514822 11708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:15:20.514886 11708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:15:20.519357 11708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:15:20.519425 11708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:15:20.519511 11708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:15:20.525055 11708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:49937/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:15:20.526180 11708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:15:20.527297 11708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:15:20.538586 11714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:15:20.539052 11717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:15:20.539738 11715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:15:20.546587 11708 server_base.cc:1034] running on GCE node
I20260924 04:15:20.547075 11708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:15:20.547840 11708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:15:20.549029 11708 hybrid_clock.cc:648] HybridClock initialized: now 1790223320548991 us; error 46 us; skew 500 ppm
Sep 24 04:15:20 dist-test-slave-p401 krb5kdc[11151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223320, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:15:20.551797 11708 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:15:20.552732 11708 webserver.cc:460] Webserver started at http://127.0.98.1:39607/ using document root <none> and password file <none>
I20260924 04:15:20.553272 11708 fs_manager.cc:359] Metadata directory not provided
I20260924 04:15:20.553344 11708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:15:20.553560 11708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 04:15:20 dist-test-slave-p401 krb5kdc[11151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223320, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:15:20.576532 11708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ts-0/data/instance:
uuid: "dcc4dfca19964e878927b734bdd08b46"
format_stamp: "Formatted at 2026-09-24 04:15:20 on dist-test-slave-p401"
server_key: "7c3cb57f392bce72c8c621df6cd10c44"
server_key_iv: "b4fcb0944e0bb2df8bad0ee9ef49021d"
server_key_version: "kuduclusterkey@0"
I20260924 04:15:20.577301 11708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ts-0/wal/instance:
uuid: "dcc4dfca19964e878927b734bdd08b46"
format_stamp: "Formatted at 2026-09-24 04:15:20 on dist-test-slave-p401"
server_key: "7c3cb57f392bce72c8c621df6cd10c44"
server_key_iv: "b4fcb0944e0bb2df8bad0ee9ef49021d"
server_key_version: "kuduclusterkey@0"
I20260924 04:15:20.584034 11708 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260924 04:15:20.607713 11725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:15:20.609577 11708 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260924 04:15:20.609728 11708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ts-0/wal
uuid: "dcc4dfca19964e878927b734bdd08b46"
format_stamp: "Formatted at 2026-09-24 04:15:20 on dist-test-slave-p401"
server_key: "7c3cb57f392bce72c8c621df6cd10c44"
server_key_iv: "b4fcb0944e0bb2df8bad0ee9ef49021d"
server_key_version: "kuduclusterkey@0"
I20260924 04:15:20.609879 11708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:15:20.632404 11708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:15:20.636040 11708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:15:20.636246 11708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:15:20.636974 11708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:15:20.638023 11708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:15:20.638126 11708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:15:20.638266 11708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:15:20.638351 11708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:15:20.649814 11708 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34349
I20260924 04:15:20.649842 11840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34349 every 8 connection(s)
I20260924 04:15:20.651086 11708 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:15:20.653623 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 11708
I20260924 04:15:20.653744 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:15:20.654019 392 external_mini_cluster.cc:1428] Setting key 56169f551301e458e2ec0bf546fb266e
Sep 24 04:15:20 dist-test-slave-p401 krb5kdc[11151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223320, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:15:20.669587 11844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:15:20.653715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40463 (local address 127.0.98.1:60979)
0924 04:15:20.655267 (+ 1552us) negotiation.cc:107] Waiting for socket to connect
0924 04:15:20.655332 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 04:15:20.656312 (+ 980us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:15:20.657048 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:15:20.657058 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:15:20.657404 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:15:20.658046 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:15:20.658066 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:15:20.659219 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:15:20.659223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:15:20.660615 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:15:20.660625 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:15:20.661411 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:15:20.661416 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:15:20.661692 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:15:20.662347 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:15:20.662367 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:15:20.664063 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:15:20.666289 (+ 2226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:15:20.666295 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:15:20.666312 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:15:20.666648 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:15:20.666971 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:15:20.666974 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:15:20.666976 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:15:20.667153 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:15:20.667646 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:15:20.667652 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:15:20.667993 (+ 341us) client_negotiation.cc:769] Sending connection context
0924 04:15:20.668181 (+ 188us) client_negotiation.cc:240] Negotiation successful
0924 04:15:20.668507 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1309,"thread_start_us":132,"threads_started":1}
I20260924 04:15:20.670336 11843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:15:20.653228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60979 (local address 127.0.98.62:40463)
0924 04:15:20.655421 (+ 2193us) server_negotiation.cc:206] Beginning negotiation
0924 04:15:20.655430 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 04:15:20.655448 (+ 18us) server_negotiation.cc:407] Connection header received
0924 04:15:20.656627 (+ 1179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:15:20.656631 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:15:20.656680 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:15:20.656755 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:15:20.658208 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:15:20.659078 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:15:20.660749 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:15:20.661094 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:15:20.661197 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:15:20.664250 (+ 3053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:15:20.664254 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:15:20.664257 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:15:20.664285 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:15:20.666158 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:15:20.666762 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:15:20.666766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:15:20.666768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:15:20.666873 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:15:20.667281 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:15:20.667285 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:15:20.667287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:15:20.667505 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:15:20.667593 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 04:15:20.670046 (+ 2453us) server_negotiation.cc:299] Negotiation successful
0924 04:15:20.670174 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2115,"thread_start_us":99,"threads_started":1}
I20260924 04:15:20.672303 11841 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40463
I20260924 04:15:20.672607 11841 heartbeater.cc:464] Registering TS with master...
I20260924 04:15:20.673209 11841 heartbeater.cc:511] Master 127.0.98.62:40463 requested a full tablet report, sending...
I20260924 04:15:20.676445 11627 ts_manager.cc:194] Registered new tserver with Master: dcc4dfca19964e878927b734bdd08b46 (127.0.98.1:34349)
I20260924 04:15:20.678099 11627 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60979
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:15:20.699965 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:49937/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40463
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 04:15:20 dist-test-slave-p401 krb5kdc[11151](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790223320, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 04:15:20.825486 11847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:15:20.825768 11847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:15:20.825838 11847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:15:20.830168 11847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:15:20.830227 11847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:15:20.830310 11847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:15:20.835613 11847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:49937/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:15:20.836674 11847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:15:20.837888 11847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:15:20.849102 11853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:15:20.849305 11854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:15:20.849165 11856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:15:20.849598 11847 server_base.cc:1034] running on GCE node
I20260924 04:15:20.849977 11847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:15:20.850626 11847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:15:20.851783 11847 hybrid_clock.cc:648] HybridClock initialized: now 1790223320851760 us; error 35 us; skew 500 ppm
Sep 24 04:15:20 dist-test-slave-p401 krb5kdc[11151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223320, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:15:20.854357 11847 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:15:20.855262 11847 webserver.cc:460] Webserver started at http://127.0.98.2:40115/ using document root <none> and password file <none>
I20260924 04:15:20.855867 11847 fs_manager.cc:359] Metadata directory not provided
I20260924 04:15:20.855932 11847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:15:20.856148 11847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 04:15:20 dist-test-slave-p401 krb5kdc[11151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223320, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:15:20.869267 11847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ts-1/data/instance:
uuid: "a848ee329a3c4497ac2f682e2c05dc73"
format_stamp: "Formatted at 2026-09-24 04:15:20 on dist-test-slave-p401"
server_key: "fd703ec751f5b54a8ade6524721be545"
server_key_iv: "803b030949eeb53c4b030304af03cbe6"
server_key_version: "kuduclusterkey@0"
I20260924 04:15:20.870034 11847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a848ee329a3c4497ac2f682e2c05dc73"
format_stamp: "Formatted at 2026-09-24 04:15:20 on dist-test-slave-p401"
server_key: "fd703ec751f5b54a8ade6524721be545"
server_key_iv: "803b030949eeb53c4b030304af03cbe6"
server_key_version: "kuduclusterkey@0"
I20260924 04:15:20.874624 11847 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260924 04:15:20.899137 11863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:15:20.905057 11847 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.002s sys 0.000s
I20260924 04:15:20.905242 11847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ts-1/wal
uuid: "a848ee329a3c4497ac2f682e2c05dc73"
format_stamp: "Formatted at 2026-09-24 04:15:20 on dist-test-slave-p401"
server_key: "fd703ec751f5b54a8ade6524721be545"
server_key_iv: "803b030949eeb53c4b030304af03cbe6"
server_key_version: "kuduclusterkey@0"
I20260924 04:15:20.905427 11847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:15:20.926498 11847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:15:20.930176 11847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:15:20.930369 11847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:15:20.931083 11847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:15:20.932220 11847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:15:20.932279 11847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:15:20.932331 11847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:15:20.932403 11847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:15:20.944612 11847 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46509
I20260924 04:15:20.945833 11847 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:15:20.946471 11976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46509 every 8 connection(s)
I20260924 04:15:20.951790 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 11847
I20260924 04:15:20.951923 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:15:20.952211 392 external_mini_cluster.cc:1428] Setting key d75a14ed7bdf9f60a0f44f0e5831cf6f
Sep 24 04:15:20 dist-test-slave-p401 krb5kdc[11151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223320, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:15:20.962913 11843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:15:20.947933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35585 (local address 127.0.98.62:40463)
0924 04:15:20.948064 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 04:15:20.948068 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:15:20.949726 (+ 1658us) server_negotiation.cc:407] Connection header received
0924 04:15:20.950975 (+ 1249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:15:20.950979 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:15:20.951035 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:15:20.951136 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:15:20.952751 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:15:20.953524 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:15:20.955346 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:15:20.955697 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:15:20.955782 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:15:20.958788 (+ 3006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:15:20.958792 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:15:20.958795 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:15:20.958823 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:15:20.960525 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:15:20.961136 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:15:20.961140 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:15:20.961142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:15:20.961227 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:15:20.961609 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:15:20.961612 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:15:20.961614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:15:20.961828 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:15:20.961912 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 04:15:20.962598 (+ 686us) server_negotiation.cc:299] Negotiation successful
0924 04:15:20.962731 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:15:20.964605 11979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:15:20.948432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40463 (local address 127.0.98.2:35585)
0924 04:15:20.949563 (+ 1131us) negotiation.cc:107] Waiting for socket to connect
0924 04:15:20.949616 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 04:15:20.950729 (+ 1113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:15:20.951416 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:15:20.951426 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:15:20.951789 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:15:20.952473 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:15:20.952495 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:15:20.953748 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:15:20.953751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:15:20.955173 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:15:20.955185 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:15:20.955988 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:15:20.955992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:15:20.956207 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:15:20.956866 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:15:20.956887 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:15:20.958598 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:15:20.960664 (+ 2066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:15:20.960669 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:15:20.960686 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:15:20.961024 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:15:20.961331 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:15:20.961333 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:15:20.961336 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:15:20.961505 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:15:20.961990 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:15:20.961997 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:15:20.962321 (+ 324us) client_negotiation.cc:769] Sending connection context
0924 04:15:20.963248 (+ 927us) client_negotiation.cc:240] Negotiation successful
0924 04:15:20.963594 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":926,"thread_start_us":153,"threads_started":1}
I20260924 04:15:20.966074 11977 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40463
I20260924 04:15:20.966413 11977 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 04:15:20.967101 11977 heartbeater.cc:511] Master 127.0.98.62:40463 requested a full tablet report, sending...
I20260924 04:15:20.970415 11627 ts_manager.cc:194] Registered new tserver with Master: a848ee329a3c4497ac2f682e2c05dc73 (127.0.98.2:46509)
I20260924 04:15:20.971149 11627 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35585
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:15:20.988560 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:49937/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40463
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:15:21.137995 11982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:15:21.138377 11982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:15:21.138509 11982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:15:21.144726 11982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:15:21.144861 11982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:15:21.144976 11982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:15:21.153374 11982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:49937/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:15:21.154969 11982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:15:21.156387 11982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:15:21.169152 11990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:15:21.169447 11987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:15:21.170528 11982 server_base.cc:1034] running on GCE node
W20260924 04:15:21.172240 11988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:15:21.172956 11982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:15:21.173735 11982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:15:21.175093 11982 hybrid_clock.cc:648] HybridClock initialized: now 1790223321174958 us; error 152 us; skew 500 ppm
Sep 24 04:15:21 dist-test-slave-p401 krb5kdc[11151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223320, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 04:15:21 dist-test-slave-p401 krb5kdc[11151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223321, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:15:21.178524 11982 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:15:21.179687 11982 webserver.cc:460] Webserver started at http://127.0.98.3:41713/ using document root <none> and password file <none>
I20260924 04:15:21.180434 11982 fs_manager.cc:359] Metadata directory not provided
I20260924 04:15:21.180577 11982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:15:21.180850 11982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 04:15:21 dist-test-slave-p401 krb5kdc[11151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223321, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:15:21.198693 11982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ts-2/data/instance:
uuid: "2700c36d9eb247ca8fb38a6d277c18c0"
format_stamp: "Formatted at 2026-09-24 04:15:21 on dist-test-slave-p401"
server_key: "4810e9e40560242d991cff89e46fbac1"
server_key_iv: "5f3699ee9579d8cbea455d639b67a837"
server_key_version: "kuduclusterkey@0"
I20260924 04:15:21.199595 11982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2700c36d9eb247ca8fb38a6d277c18c0"
format_stamp: "Formatted at 2026-09-24 04:15:21 on dist-test-slave-p401"
server_key: "4810e9e40560242d991cff89e46fbac1"
server_key_iv: "5f3699ee9579d8cbea455d639b67a837"
server_key_version: "kuduclusterkey@0"
I20260924 04:15:21.206919 11982 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.000s
I20260924 04:15:21.233927 11997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:15:21.236085 11982 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.004s sys 0.000s
I20260924 04:15:21.236238 11982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ts-2/wal
uuid: "2700c36d9eb247ca8fb38a6d277c18c0"
format_stamp: "Formatted at 2026-09-24 04:15:21 on dist-test-slave-p401"
server_key: "4810e9e40560242d991cff89e46fbac1"
server_key_iv: "5f3699ee9579d8cbea455d639b67a837"
server_key_version: "kuduclusterkey@0"
I20260924 04:15:21.236387 11982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:15:21.256697 11982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:15:21.260242 11982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:15:21.260483 11982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:15:21.261197 11982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:15:21.262296 11982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:15:21.262420 11982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:15:21.262509 11982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:15:21.262557 11982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:15:21.272790 11982 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43979
I20260924 04:15:21.273962 11982 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:15:21.276120 12110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43979 every 8 connection(s)
I20260924 04:15:21.280349 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 11982
I20260924 04:15:21.280479 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:15:21.280752 392 external_mini_cluster.cc:1428] Setting key 623ac3ce2f4a0e07b336d5a3ce4590eb
Sep 24 04:15:21 dist-test-slave-p401 krb5kdc[11151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223321, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:15:21.295495 11843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:15:21.279792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54907 (local address 127.0.98.62:40463)
0924 04:15:21.279909 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 04:15:21.279914 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:15:21.280427 (+ 513us) server_negotiation.cc:407] Connection header received
0924 04:15:21.283719 (+ 3292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:15:21.283722 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:15:21.283780 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:15:21.283832 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:15:21.285508 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:15:21.286362 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:15:21.288058 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:15:21.288403 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:15:21.288504 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:15:21.291355 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:15:21.291359 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:15:21.291363 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:15:21.291392 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:15:21.293047 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:15:21.293665 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:15:21.293669 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:15:21.293671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:15:21.293762 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:15:21.294153 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:15:21.294156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:15:21.294158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:15:21.294381 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:15:21.294465 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 04:15:21.295182 (+ 717us) server_negotiation.cc:299] Negotiation successful
0924 04:15:21.295331 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 04:15:21.296490 12113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:15:21.278666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40463 (local address 127.0.98.3:54907)
0924 04:15:21.280262 (+ 1596us) negotiation.cc:107] Waiting for socket to connect
0924 04:15:21.280318 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 04:15:21.283443 (+ 3125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:15:21.284260 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:15:21.284271 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:15:21.284618 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:15:21.285312 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:15:21.285334 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:15:21.286515 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:15:21.286519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:15:21.287941 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:15:21.287951 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:15:21.288562 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:15:21.288566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:15:21.288759 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:15:21.289448 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:15:21.289477 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:15:21.291142 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:15:21.293176 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:15:21.293182 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:15:21.293201 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:15:21.293545 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:15:21.293872 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:15:21.293875 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:15:21.293877 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:15:21.294053 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:15:21.294530 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:15:21.294536 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:15:21.294885 (+ 349us) client_negotiation.cc:769] Sending connection context
0924 04:15:21.295090 (+ 205us) client_negotiation.cc:240] Negotiation successful
0924 04:15:21.295458 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1374,"thread_start_us":125,"threads_started":1}
I20260924 04:15:21.297693 12111 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40463
I20260924 04:15:21.297919 12111 heartbeater.cc:464] Registering TS with master...
I20260924 04:15:21.298465 12111 heartbeater.cc:511] Master 127.0.98.62:40463 requested a full tablet report, sending...
I20260924 04:15:21.299528 11627 ts_manager.cc:194] Registered new tserver with Master: 2700c36d9eb247ca8fb38a6d277c18c0 (127.0.98.3:43979)
I20260924 04:15:21.300225 11627 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54907
I20260924 04:15:21.307668 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:15:21.319521 11843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:15:21.310088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33282 (local address 127.0.98.62:40463)
0924 04:15:21.310223 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 04:15:21.310227 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:15:21.310422 (+ 195us) server_negotiation.cc:407] Connection header received
0924 04:15:21.310553 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:15:21.310556 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:15:21.310608 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:15:21.310694 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:15:21.311577 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:15:21.312412 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:15:21.313588 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:15:21.313913 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:15:21.314070 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:15:21.315591 (+ 1521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:15:21.315595 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:15:21.315598 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:15:21.315625 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:15:21.317397 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:15:21.317841 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:15:21.317845 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:15:21.317847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:15:21.317935 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:15:21.318264 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:15:21.318268 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:15:21.318269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:15:21.318476 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:15:21.318597 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 04:15:21.319248 (+ 651us) server_negotiation.cc:299] Negotiation successful
0924 04:15:21.319364 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 04:15:21.322654 11627 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33282:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:15:21.327116 11627 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:15:21.343528 12125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:15:21.336816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46340 (local address 127.0.98.2:46509)
0924 04:15:21.337846 (+ 1030us) server_negotiation.cc:206] Beginning negotiation
0924 04:15:21.337858 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:15:21.337882 (+ 24us) server_negotiation.cc:407] Connection header received
0924 04:15:21.337959 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:15:21.337965 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:15:21.338151 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:15:21.338287 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:15:21.338543 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:15:21.339408 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:15:21.341670 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:15:21.342574 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:15:21.342672 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:15:21.342825 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 04:15:21.343263 (+ 438us) server_negotiation.cc:299] Negotiation successful
0924 04:15:21.343367 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":928,"thread_start_us":93,"threads_started":1}
I20260924 04:15:21.344122 12126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:15:21.336624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46509 (local address 127.0.0.1:46340)
0924 04:15:21.337279 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0924 04:15:21.337291 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:15:21.337370 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:15:21.338292 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:15:21.338298 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:15:21.338319 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:15:21.338437 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:15:21.338443 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:15:21.339507 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:15:21.339510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:15:21.341509 (+ 1999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:15:21.341520 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:15:21.342889 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:15:21.342893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:15:21.343048 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:15:21.343080 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 04:15:21.343184 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 04:15:21.344016 (+ 832us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":83,"threads_started":1}
I20260924 04:15:21.347504 11911 tablet_service.cc:1467] Processing CreateTablet for tablet 3aa5a01f4078404b9f0510c2446706a1 (DEFAULT_TABLE table=test-table [id=5d4bb39b4ab8412a9c7f37481beb5fee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:15:21.348361 12123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:15:21.335881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43979 (local address 127.0.0.1:35984)
0924 04:15:21.336380 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0924 04:15:21.336424 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 04:15:21.336541 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:15:21.343971 (+ 7430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:15:21.343974 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:15:21.343996 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:15:21.344110 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:15:21.344117 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:15:21.345720 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:15:21.345724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:15:21.346925 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:15:21.346935 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:15:21.347977 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:15:21.347981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:15:21.348089 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:15:21.348113 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 04:15:21.348157 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 04:15:21.348220 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":116,"threads_started":1}
I20260924 04:15:21.348726 11911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3aa5a01f4078404b9f0510c2446706a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:15:21.349016 12127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:15:21.337293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35984 (local address 127.0.98.3:43979)
0924 04:15:21.343452 (+ 6159us) server_negotiation.cc:206] Beginning negotiation
0924 04:15:21.343464 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:15:21.343490 (+ 26us) server_negotiation.cc:407] Connection header received
0924 04:15:21.343566 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:15:21.343571 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:15:21.343795 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:15:21.344272 (+ 477us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:15:21.344422 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:15:21.345587 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:15:21.347093 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:15:21.347852 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:15:21.348602 (+ 750us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:15:21.348744 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 04:15:21.348793 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 04:15:21.348878 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6044,"thread_start_us":156,"threads_started":1}
I20260924 04:15:21.351882 12045 tablet_service.cc:1467] Processing CreateTablet for tablet 3aa5a01f4078404b9f0510c2446706a1 (DEFAULT_TABLE table=test-table [id=5d4bb39b4ab8412a9c7f37481beb5fee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:15:21.352907 12045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3aa5a01f4078404b9f0510c2446706a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:15:21.361152 12128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:15:21.337688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49468 (local address 127.0.98.1:34349)
0924 04:15:21.347307 (+ 9619us) server_negotiation.cc:206] Beginning negotiation
0924 04:15:21.347314 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:15:21.347338 (+ 24us) server_negotiation.cc:407] Connection header received
0924 04:15:21.347407 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:15:21.347412 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:15:21.347600 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:15:21.347743 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:15:21.356883 (+ 9140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:15:21.357982 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:15:21.359490 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:15:21.360313 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:15:21.360393 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:15:21.360522 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 04:15:21.360905 (+ 383us) server_negotiation.cc:299] Negotiation successful
0924 04:15:21.361008 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9484,"thread_start_us":130,"threads_started":1}
I20260924 04:15:21.361256 12129 tablet_bootstrap.cc:492] T 3aa5a01f4078404b9f0510c2446706a1 P 2700c36d9eb247ca8fb38a6d277c18c0: Bootstrap starting.
I20260924 04:15:21.361760 12124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:15:21.336895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34349 (local address 127.0.0.1:49468)
0924 04:15:21.337533 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0924 04:15:21.337544 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:15:21.337609 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:15:21.356558 (+ 18949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:15:21.356562 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:15:21.356585 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:15:21.356715 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:15:21.356722 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:15:21.358132 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:15:21.358135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:15:21.359344 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:15:21.359355 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:15:21.360617 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:15:21.360620 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:15:21.360734 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:15:21.360761 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 04:15:21.361534 (+ 773us) client_negotiation.cc:240] Negotiation successful
0924 04:15:21.361612 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":66,"threads_started":1}
I20260924 04:15:21.364034 12130 tablet_bootstrap.cc:492] T 3aa5a01f4078404b9f0510c2446706a1 P a848ee329a3c4497ac2f682e2c05dc73: Bootstrap starting.
I20260924 04:15:21.364786 11770 tablet_service.cc:1467] Processing CreateTablet for tablet 3aa5a01f4078404b9f0510c2446706a1 (DEFAULT_TABLE table=test-table [id=5d4bb39b4ab8412a9c7f37481beb5fee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:15:21.365850 11770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3aa5a01f4078404b9f0510c2446706a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:15:21.366814 12130 tablet_bootstrap.cc:654] T 3aa5a01f4078404b9f0510c2446706a1 P a848ee329a3c4497ac2f682e2c05dc73: Neither blocks nor log segments found. Creating new log.
I20260924 04:15:21.367818 12130 log.cc:824] T 3aa5a01f4078404b9f0510c2446706a1 P a848ee329a3c4497ac2f682e2c05dc73: Log is configured to *not* fsync() on all Append() calls
I20260924 04:15:21.369935 12130 tablet_bootstrap.cc:492] T 3aa5a01f4078404b9f0510c2446706a1 P a848ee329a3c4497ac2f682e2c05dc73: No bootstrap required, opened a new log
I20260924 04:15:21.370114 12130 ts_tablet_manager.cc:1397] T 3aa5a01f4078404b9f0510c2446706a1 P a848ee329a3c4497ac2f682e2c05dc73: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:15:21.373940 12130 raft_consensus.cc:348] T 3aa5a01f4078404b9f0510c2446706a1 P a848ee329a3c4497ac2f682e2c05dc73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2700c36d9eb247ca8fb38a6d277c18c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43979 } } peers { permanent_uuid: "a848ee329a3c4497ac2f682e2c05dc73" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46509 } } peers { permanent_uuid: "dcc4dfca19964e878927b734bdd08b46" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34349 } }
I20260924 04:15:21.374310 12130 raft_consensus.cc:374] T 3aa5a01f4078404b9f0510c2446706a1 P a848ee329a3c4497ac2f682e2c05dc73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:15:21.374406 12130 raft_consensus.cc:729] T 3aa5a01f4078404b9f0510c2446706a1 P a848ee329a3c4497ac2f682e2c05dc73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a848ee329a3c4497ac2f682e2c05dc73, State: Initialized, Role: FOLLOWER
I20260924 04:15:21.374675 12129 tablet_bootstrap.cc:654] T 3aa5a01f4078404b9f0510c2446706a1 P 2700c36d9eb247ca8fb38a6d277c18c0: Neither blocks nor log segments found. Creating new log.
I20260924 04:15:21.374956 12130 consensus_queue.cc:260] T 3aa5a01f4078404b9f0510c2446706a1 P a848ee329a3c4497ac2f682e2c05dc73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2700c36d9eb247ca8fb38a6d277c18c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43979 } } peers { permanent_uuid: "a848ee329a3c4497ac2f682e2c05dc73" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46509 } } peers { permanent_uuid: "dcc4dfca19964e878927b734bdd08b46" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34349 } }
I20260924 04:15:21.375592 12129 log.cc:824] T 3aa5a01f4078404b9f0510c2446706a1 P 2700c36d9eb247ca8fb38a6d277c18c0: Log is configured to *not* fsync() on all Append() calls
I20260924 04:15:21.376066 12130 ts_tablet_manager.cc:1428] T 3aa5a01f4078404b9f0510c2446706a1 P a848ee329a3c4497ac2f682e2c05dc73: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 04:15:21.376423 11977 heartbeater.cc:503] Master 127.0.98.62:40463 was elected leader, sending a full tablet report...
I20260924 04:15:21.377990 12132 tablet_bootstrap.cc:492] T 3aa5a01f4078404b9f0510c2446706a1 P dcc4dfca19964e878927b734bdd08b46: Bootstrap starting.
I20260924 04:15:21.380287 12132 tablet_bootstrap.cc:654] T 3aa5a01f4078404b9f0510c2446706a1 P dcc4dfca19964e878927b734bdd08b46: Neither blocks nor log segments found. Creating new log.
I20260924 04:15:21.381276 12132 log.cc:824] T 3aa5a01f4078404b9f0510c2446706a1 P dcc4dfca19964e878927b734bdd08b46: Log is configured to *not* fsync() on all Append() calls
I20260924 04:15:21.383812 12132 tablet_bootstrap.cc:492] T 3aa5a01f4078404b9f0510c2446706a1 P dcc4dfca19964e878927b734bdd08b46: No bootstrap required, opened a new log
I20260924 04:15:21.383810 12129 tablet_bootstrap.cc:492] T 3aa5a01f4078404b9f0510c2446706a1 P 2700c36d9eb247ca8fb38a6d277c18c0: No bootstrap required, opened a new log
I20260924 04:15:21.384012 12129 ts_tablet_manager.cc:1397] T 3aa5a01f4078404b9f0510c2446706a1 P 2700c36d9eb247ca8fb38a6d277c18c0: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.000s
I20260924 04:15:21.384014 12132 ts_tablet_manager.cc:1397] T 3aa5a01f4078404b9f0510c2446706a1 P dcc4dfca19964e878927b734bdd08b46: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:15:21.388175 12129 raft_consensus.cc:348] T 3aa5a01f4078404b9f0510c2446706a1 P 2700c36d9eb247ca8fb38a6d277c18c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2700c36d9eb247ca8fb38a6d277c18c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43979 } } peers { permanent_uuid: "a848ee329a3c4497ac2f682e2c05dc73" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46509 } } peers { permanent_uuid: "dcc4dfca19964e878927b734bdd08b46" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34349 } }
I20260924 04:15:21.388468 12129 raft_consensus.cc:374] T 3aa5a01f4078404b9f0510c2446706a1 P 2700c36d9eb247ca8fb38a6d277c18c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:15:21.388518 12129 raft_consensus.cc:729] T 3aa5a01f4078404b9f0510c2446706a1 P 2700c36d9eb247ca8fb38a6d277c18c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2700c36d9eb247ca8fb38a6d277c18c0, State: Initialized, Role: FOLLOWER
I20260924 04:15:21.388496 12132 raft_consensus.cc:348] T 3aa5a01f4078404b9f0510c2446706a1 P dcc4dfca19964e878927b734bdd08b46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2700c36d9eb247ca8fb38a6d277c18c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43979 } } peers { permanent_uuid: "a848ee329a3c4497ac2f682e2c05dc73" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46509 } } peers { permanent_uuid: "dcc4dfca19964e878927b734bdd08b46" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34349 } }
I20260924 04:15:21.388751 12132 raft_consensus.cc:374] T 3aa5a01f4078404b9f0510c2446706a1 P dcc4dfca19964e878927b734bdd08b46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:15:21.388798 12132 raft_consensus.cc:729] T 3aa5a01f4078404b9f0510c2446706a1 P dcc4dfca19964e878927b734bdd08b46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcc4dfca19964e878927b734bdd08b46, State: Initialized, Role: FOLLOWER
I20260924 04:15:21.389067 12129 consensus_queue.cc:260] T 3aa5a01f4078404b9f0510c2446706a1 P 2700c36d9eb247ca8fb38a6d277c18c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2700c36d9eb247ca8fb38a6d277c18c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43979 } } peers { permanent_uuid: "a848ee329a3c4497ac2f682e2c05dc73" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46509 } } peers { permanent_uuid: "dcc4dfca19964e878927b734bdd08b46" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34349 } }
I20260924 04:15:21.389436 12132 consensus_queue.cc:260] T 3aa5a01f4078404b9f0510c2446706a1 P dcc4dfca19964e878927b734bdd08b46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2700c36d9eb247ca8fb38a6d277c18c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43979 } } peers { permanent_uuid: "a848ee329a3c4497ac2f682e2c05dc73" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46509 } } peers { permanent_uuid: "dcc4dfca19964e878927b734bdd08b46" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34349 } }
I20260924 04:15:21.390290 12129 ts_tablet_manager.cc:1428] T 3aa5a01f4078404b9f0510c2446706a1 P 2700c36d9eb247ca8fb38a6d277c18c0: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 04:15:21.390504 12132 ts_tablet_manager.cc:1428] T 3aa5a01f4078404b9f0510c2446706a1 P dcc4dfca19964e878927b734bdd08b46: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 04:15:21.390707 12111 heartbeater.cc:503] Master 127.0.98.62:40463 was elected leader, sending a full tablet report...
I20260924 04:15:21.391458 11841 heartbeater.cc:503] Master 127.0.98.62:40463 was elected leader, sending a full tablet report...
W20260924 04:15:21.402952 11842 tablet.cc:2369] T 3aa5a01f4078404b9f0510c2446706a1 P dcc4dfca19964e878927b734bdd08b46: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:15:21.445492 12136 raft_consensus.cc:482] T 3aa5a01f4078404b9f0510c2446706a1 P 2700c36d9eb247ca8fb38a6d277c18c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:15:21.445744 12136 raft_consensus.cc:504] T 3aa5a01f4078404b9f0510c2446706a1 P 2700c36d9eb247ca8fb38a6d277c18c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2700c36d9eb247ca8fb38a6d277c18c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43979 } } peers { permanent_uuid: "a848ee329a3c4497ac2f682e2c05dc73" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46509 } } peers { permanent_uuid: "dcc4dfca19964e878927b734bdd08b46" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34349 } }
W20260924 04:15:21.447387 11978 tablet.cc:2369] T 3aa5a01f4078404b9f0510c2446706a1 P a848ee329a3c4497ac2f682e2c05dc73: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:15:21.447206 12136 leader_election.cc:288] T 3aa5a01f4078404b9f0510c2446706a1 P 2700c36d9eb247ca8fb38a6d277c18c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a848ee329a3c4497ac2f682e2c05dc73 (127.0.98.2:46509), dcc4dfca19964e878927b734bdd08b46 (127.0.98.1:34349)
I20260924 04:15:21.453873 12113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:15:21.448321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46509 (local address 127.0.98.3:47777)
0924 04:15:21.449030 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0924 04:15:21.449048 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 04:15:21.449169 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:15:21.449759 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:15:21.449762 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:15:21.449780 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:15:21.449906 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:15:21.449913 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:15:21.451027 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:15:21.451030 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:15:21.452354 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:15:21.452366 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:15:21.453441 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:15:21.453445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:15:21.453547 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:15:21.453570 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 04:15:21.453639 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 04:15:21.453709 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622}
I20260924 04:15:21.454891 12125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:15:21.449013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47777 (local address 127.0.98.2:46509)
0924 04:15:21.449140 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 04:15:21.449144 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:15:21.449161 (+ 17us) server_negotiation.cc:407] Connection header received
0924 04:15:21.449269 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:15:21.449272 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:15:21.449318 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:15:21.449423 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:15:21.450013 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:15:21.450865 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:15:21.452514 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:15:21.453188 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:15:21.453289 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:15:21.453374 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 04:15:21.454656 (+ 1282us) server_negotiation.cc:299] Negotiation successful
0924 04:15:21.454755 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 04:15:21.455554 11913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3aa5a01f4078404b9f0510c2446706a1" candidate_uuid: "2700c36d9eb247ca8fb38a6d277c18c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a848ee329a3c4497ac2f682e2c05dc73" is_pre_election: true
I20260924 04:15:21.455809 12128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:15:21.449584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45535 (local address 127.0.98.1:34349)
0924 04:15:21.449699 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 04:15:21.449703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:15:21.449714 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:15:21.449759 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:15:21.449762 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:15:21.449799 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:15:21.449868 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:15:21.451200 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:15:21.451970 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:15:21.453484 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:15:21.454105 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:15:21.454194 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:15:21.454261 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 04:15:21.455593 (+ 1332us) server_negotiation.cc:299] Negotiation successful
0924 04:15:21.455683 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 04:15:21.455915 11913 raft_consensus.cc:2454] T 3aa5a01f4078404b9f0510c2446706a1 P a848ee329a3c4497ac2f682e2c05dc73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2700c36d9eb247ca8fb38a6d277c18c0 in term 0.
I20260924 04:15:21.456200 12138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:15:21.448617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34349 (local address 127.0.98.3:45535)
0924 04:15:21.449520 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0924 04:15:21.449535 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:15:21.449628 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:15:21.450868 (+ 1240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:15:21.450872 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:15:21.450893 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:15:21.451018 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:15:21.451026 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:15:21.452180 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:15:21.452184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:15:21.453330 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:15:21.453342 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:15:21.454377 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:15:21.454381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:15:21.454492 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:15:21.454517 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 04:15:21.455944 (+ 1427us) client_negotiation.cc:240] Negotiation successful
0924 04:15:21.456023 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":839,"thread_start_us":103,"threads_started":1}
I20260924 04:15:21.456920 11774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3aa5a01f4078404b9f0510c2446706a1" candidate_uuid: "2700c36d9eb247ca8fb38a6d277c18c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcc4dfca19964e878927b734bdd08b46" is_pre_election: true
I20260924 04:15:21.457183 11774 raft_consensus.cc:2454] T 3aa5a01f4078404b9f0510c2446706a1 P dcc4dfca19964e878927b734bdd08b46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2700c36d9eb247ca8fb38a6d277c18c0 in term 0.
I20260924 04:15:21.458024 11999 leader_election.cc:302] T 3aa5a01f4078404b9f0510c2446706a1 P 2700c36d9eb247ca8fb38a6d277c18c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2700c36d9eb247ca8fb38a6d277c18c0, a848ee329a3c4497ac2f682e2c05dc73; no voters:
I20260924 04:15:21.458765 12136 raft_consensus.cc:2781] T 3aa5a01f4078404b9f0510c2446706a1 P 2700c36d9eb247ca8fb38a6d277c18c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:15:21.458916 12136 raft_consensus.cc:482] T 3aa5a01f4078404b9f0510c2446706a1 P 2700c36d9eb247ca8fb38a6d277c18c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:15:21.459021 12136 raft_consensus.cc:3037] T 3aa5a01f4078404b9f0510c2446706a1 P 2700c36d9eb247ca8fb38a6d277c18c0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:15:21.460420 12136 raft_consensus.cc:504] T 3aa5a01f4078404b9f0510c2446706a1 P 2700c36d9eb247ca8fb38a6d277c18c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2700c36d9eb247ca8fb38a6d277c18c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43979 } } peers { permanent_uuid: "a848ee329a3c4497ac2f682e2c05dc73" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46509 } } peers { permanent_uuid: "dcc4dfca19964e878927b734bdd08b46" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34349 } }
I20260924 04:15:21.460796 12136 leader_election.cc:288] T 3aa5a01f4078404b9f0510c2446706a1 P 2700c36d9eb247ca8fb38a6d277c18c0 [CANDIDATE]: Term 1 election: Requested vote from peers a848ee329a3c4497ac2f682e2c05dc73 (127.0.98.2:46509), dcc4dfca19964e878927b734bdd08b46 (127.0.98.1:34349)
I20260924 04:15:21.461193 11913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3aa5a01f4078404b9f0510c2446706a1" candidate_uuid: "2700c36d9eb247ca8fb38a6d277c18c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a848ee329a3c4497ac2f682e2c05dc73"
I20260924 04:15:21.461262 11774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3aa5a01f4078404b9f0510c2446706a1" candidate_uuid: "2700c36d9eb247ca8fb38a6d277c18c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcc4dfca19964e878927b734bdd08b46"
I20260924 04:15:21.461308 11913 raft_consensus.cc:3037] T 3aa5a01f4078404b9f0510c2446706a1 P a848ee329a3c4497ac2f682e2c05dc73 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:15:21.461355 11774 raft_consensus.cc:3037] T 3aa5a01f4078404b9f0510c2446706a1 P dcc4dfca19964e878927b734bdd08b46 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:15:21.462498 11913 raft_consensus.cc:2454] T 3aa5a01f4078404b9f0510c2446706a1 P a848ee329a3c4497ac2f682e2c05dc73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2700c36d9eb247ca8fb38a6d277c18c0 in term 1.
I20260924 04:15:21.462498 11774 raft_consensus.cc:2454] T 3aa5a01f4078404b9f0510c2446706a1 P dcc4dfca19964e878927b734bdd08b46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2700c36d9eb247ca8fb38a6d277c18c0 in term 1.
I20260924 04:15:21.462781 11999 leader_election.cc:302] T 3aa5a01f4078404b9f0510c2446706a1 P 2700c36d9eb247ca8fb38a6d277c18c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2700c36d9eb247ca8fb38a6d277c18c0, a848ee329a3c4497ac2f682e2c05dc73; no voters:
I20260924 04:15:21.463027 12136 raft_consensus.cc:2781] T 3aa5a01f4078404b9f0510c2446706a1 P 2700c36d9eb247ca8fb38a6d277c18c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:15:21.463248 12136 raft_consensus.cc:686] T 3aa5a01f4078404b9f0510c2446706a1 P 2700c36d9eb247ca8fb38a6d277c18c0 [term 1 LEADER]: Becoming Leader. State: Replica: 2700c36d9eb247ca8fb38a6d277c18c0, State: Running, Role: LEADER
I20260924 04:15:21.466068 12136 consensus_queue.cc:237] T 3aa5a01f4078404b9f0510c2446706a1 P 2700c36d9eb247ca8fb38a6d277c18c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2700c36d9eb247ca8fb38a6d277c18c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43979 } } peers { permanent_uuid: "a848ee329a3c4497ac2f682e2c05dc73" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46509 } } peers { permanent_uuid: "dcc4dfca19964e878927b734bdd08b46" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34349 } }
I20260924 04:15:21.469401 11627 catalog_manager.cc:5495] T 3aa5a01f4078404b9f0510c2446706a1 P 2700c36d9eb247ca8fb38a6d277c18c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2700c36d9eb247ca8fb38a6d277c18c0 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "2700c36d9eb247ca8fb38a6d277c18c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2700c36d9eb247ca8fb38a6d277c18c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a848ee329a3c4497ac2f682e2c05dc73" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcc4dfca19964e878927b734bdd08b46" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34349 } health_report { overall_health: UNKNOWN } } }
Sep 24 04:15:21 dist-test-slave-p401 krb5kdc[11151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223320, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:15:21.495446 12127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:15:21.491893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36000 (local address 127.0.98.3:43979)
0924 04:15:21.492173 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0924 04:15:21.492177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:15:21.492187 (+ 10us) server_negotiation.cc:407] Connection header received
0924 04:15:21.492257 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:15:21.492259 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:15:21.492323 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:15:21.492419 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:15:21.492662 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:15:21.493160 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:15:21.494010 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:15:21.494229 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:15:21.494315 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:15:21.494614 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:15:21.495042 (+ 428us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:15:21.495153 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 04:15:21.495217 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 04:15:21.495298 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209}
I20260924 04:15:21.502615 11913 raft_consensus.cc:1261] T 3aa5a01f4078404b9f0510c2446706a1 P a848ee329a3c4497ac2f682e2c05dc73 [term 1 FOLLOWER]: Refusing update from remote peer 2700c36d9eb247ca8fb38a6d277c18c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:15:21.503592 12139 consensus_queue.cc:1035] T 3aa5a01f4078404b9f0510c2446706a1 P 2700c36d9eb247ca8fb38a6d277c18c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a848ee329a3c4497ac2f682e2c05dc73" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:15:21.503788 11774 raft_consensus.cc:1261] T 3aa5a01f4078404b9f0510c2446706a1 P dcc4dfca19964e878927b734bdd08b46 [term 1 FOLLOWER]: Refusing update from remote peer 2700c36d9eb247ca8fb38a6d277c18c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:15:21.504585 12139 consensus_queue.cc:1035] T 3aa5a01f4078404b9f0510c2446706a1 P 2700c36d9eb247ca8fb38a6d277c18c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcc4dfca19964e878927b734bdd08b46" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:15:21.518810 12144 mvcc.cc:205] Tried to move back new op lower bound from 7332754724865343488 to 7332754724727209984. Current Snapshot: MvccSnapshot[applied={T|T < 7332754724865343488}]
I20260924 04:15:21.525123 11622 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:15:21.525300 11622 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:15:21.527720 11622 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c59a33a32c8a4e08a8814e3da7a2fcb1: Sending DeleteTablet for 3 replicas of tablet 3aa5a01f4078404b9f0510c2446706a1
I20260924 04:15:21.528497 12045 tablet_service.cc:1514] Processing DeleteTablet for tablet 3aa5a01f4078404b9f0510c2446706a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:15:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35984
I20260924 04:15:21.528487 11911 tablet_service.cc:1514] Processing DeleteTablet for tablet 3aa5a01f4078404b9f0510c2446706a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:15:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46340
I20260924 04:15:21.528515 11770 tablet_service.cc:1514] Processing DeleteTablet for tablet 3aa5a01f4078404b9f0510c2446706a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:15:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49468
I20260924 04:15:21.528957 12152 tablet_replica.cc:330] T 3aa5a01f4078404b9f0510c2446706a1 P a848ee329a3c4497ac2f682e2c05dc73: stopping tablet replica
I20260924 04:15:21.529307 12152 raft_consensus.cc:2229] T 3aa5a01f4078404b9f0510c2446706a1 P a848ee329a3c4497ac2f682e2c05dc73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:15:21.529551 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 11708
I20260924 04:15:21.529604 12152 raft_consensus.cc:2258] T 3aa5a01f4078404b9f0510c2446706a1 P a848ee329a3c4497ac2f682e2c05dc73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:15:21.530184 12154 tablet_replica.cc:330] T 3aa5a01f4078404b9f0510c2446706a1 P 2700c36d9eb247ca8fb38a6d277c18c0: stopping tablet replica
I20260924 04:15:21.530431 12154 raft_consensus.cc:2229] T 3aa5a01f4078404b9f0510c2446706a1 P 2700c36d9eb247ca8fb38a6d277c18c0 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:15:21.530746 12154 raft_consensus.cc:2258] T 3aa5a01f4078404b9f0510c2446706a1 P 2700c36d9eb247ca8fb38a6d277c18c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:15:21.532433 12154 ts_tablet_manager.cc:1905] T 3aa5a01f4078404b9f0510c2446706a1 P 2700c36d9eb247ca8fb38a6d277c18c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:15:21.535143 12154 ts_tablet_manager.cc:1918] T 3aa5a01f4078404b9f0510c2446706a1 P 2700c36d9eb247ca8fb38a6d277c18c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:15:21.535282 12154 log.cc:1196] T 3aa5a01f4078404b9f0510c2446706a1 P 2700c36d9eb247ca8fb38a6d277c18c0: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790223216265354-392-0/minicluster-data/ts-2/wal/wals/3aa5a01f4078404b9f0510c2446706a1
I20260924 04:15:21.535578 12154 ts_tablet_manager.cc:1939] T 3aa5a01f4078404b9f0510c2446706a1 P 2700c36d9eb247ca8fb38a6d277c18c0: Deleting consensus metadata
W20260924 04:15:21.536478 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34349 (error 108)
I20260924 04:15:21.537101 11612 catalog_manager.cc:4841] TS 2700c36d9eb247ca8fb38a6d277c18c0 (127.0.98.3:43979): tablet 3aa5a01f4078404b9f0510c2446706a1 (table test-table [id=5d4bb39b4ab8412a9c7f37481beb5fee]) successfully deleted
I20260924 04:15:21.537622 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 11847
I20260924 04:15:21.549151 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 11982
I20260924 04:15:21.549211 12124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:15:21.548888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34349 (local address 127.0.0.1:49484)
0924 04:15:21.549027 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 04:15:21.549104 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260924 04:15:21.549450 11612 catalog_manager.cc:4584] TS dcc4dfca19964e878927b734bdd08b46 (127.0.98.1:34349): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3aa5a01f4078404b9f0510c2446706a1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34349: connect: Connection refused (error 111)
I20260924 04:15:21.550483 12124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:15:21.549316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46509 (local address 127.0.0.1:46346)
0924 04:15:21.550386 (+ 1070us) negotiation.cc:107] Waiting for socket to connect
0924 04:15:21.550428 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:46509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1009}
I20260924 04:15:21.554777 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 11595
I20260924 04:15:21.560370 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 04:15:21.905033 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 04:15:21.905182 392 mini_ranger.cc:66] Stopping Ranger...
I20260924 04:15:22.238956 392 mini_ranger.cc:68] Stopped Ranger
2026-09-24 04:15:22.239 UTC [11181] LOG: received smart shutdown request
2026-09-24 04:15:22.242 UTC [11181] LOG: background worker "logical replication launcher" (PID 11190) exited with exit code 1
2026-09-24 04:15:22.242 UTC [11185] LOG: shutting down
2026-09-24 04:15:22.417 UTC [11181] LOG: database system is shut down
2026-09-24T04:15:22Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (43941 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:15:22 dist-test-slave-p401 krb5kdc[12165](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:15:22 dist-test-slave-p401 krb5kdc[12165](info): set up 2 sockets
Sep 24 04:15:22 dist-test-slave-p401 krb5kdc[12165](info): commencing operation
krb5kdc: starting...
W20260924 04:15:23.451040 392 mini_kdc.cc:121] Time spent starting KDC: real 0.922s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/krb5kdc/test-user.keytab.
Sep 24 04:15:23 dist-test-slave-p401 krb5kdc[12165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:15:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:15:23Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260924 04:15:23.605849 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task6HV92s/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/postgres -l logfile start
2026-09-24 04:15:26.021 UTC [12195] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 04:15:26.021 UTC [12195] LOG: listening on IPv4 address "127.0.98.20", port 41603
2026-09-24 04:15:26.025 UTC [12195] LOG: listening on Unix socket "/tmp/.s.PGSQL.41603"
2026-09-24 04:15:26.054 UTC [12198] LOG: database system was shut down at 2026-09-24 04:15:24 UTC
2026-09-24 04:15:26.058 UTC [12195] LOG: database system is ready to accept connections
I20260924 04:15:26.918907 392 mini_postgres.cc:96] Postgres bound to 41603
2026-09-24 04:15:26.924 UTC [12206] FATAL: database "slave" does not exist
127.0.98.20:41603 - accepting connections
I20260924 04:15:26.925096 392 mini_ranger.cc:161] Starting Ranger...
I20260924 04:15:26.939122 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 04:15:27.415133 392 mini_ranger.cc:87] Created ranger Postgres database
I20260924 04:15:27.415253 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ranger-admin
2026-09-24 04:15:27,681 [I] DB FLAVOR :POSTGRES
2026-09-24 04:15:27,681 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 04:15:27,681 [I] Checking connection..
2026-09-24 04:15:27,681 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6HV92s/build/debug/bin/postgresql.jar:/tmp/dist-test-task6HV92s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 04:15:27,930 [I] Checking connection passed.
2026-09-24 04:15:27,931 [I] --------- Verifying version history table ---------
2026-09-24 04:15:27,931 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6HV92s/build/debug/bin/postgresql.jar:/tmp/dist-test-task6HV92s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 04:15:28,169 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 04:15:28,170 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6HV92s/build/debug/bin/postgresql.jar:/tmp/dist-test-task6HV92s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 04:15:28,395 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 04:15:28,395 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 04:15:28,395 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6HV92s/build/debug/bin/postgresql.jar:/tmp/dist-test-task6HV92s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41603/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6HV92s/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 04:15:28.632 UTC [12270] WARNING: there is no transaction in progress
2026-09-24 04:15:28,638 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 04:15:28,638 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6HV92s/build/debug/bin/postgresql.jar:/tmp/dist-test-task6HV92s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 04:15:28,868 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 04:15:28,868 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 04:15:28,868 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6HV92s/build/debug/bin/postgresql.jar:/tmp/dist-test-task6HV92s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 04:15:29,106 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6HV92s/build/debug/bin/postgresql.jar:/tmp/dist-test-task6HV92s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 04:15:29,356 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6HV92s/build/debug/bin/postgresql.jar:/tmp/dist-test-task6HV92s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-p401.c.gcp-upstream.internal','N') ;"
2026-09-24 04:15:29,585 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6HV92s/build/debug/bin/postgresql.jar:/tmp/dist-test-task6HV92s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 04:15:29,831 [I] Table x_portal_user does not exist in database ranger
2026-09-24 04:15:29,831 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6HV92s/build/debug/bin/postgresql.jar:/tmp/dist-test-task6HV92s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 04:15:30,087 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 04:15:30,088 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6HV92s/build/debug/bin/postgresql.jar:/tmp/dist-test-task6HV92s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 04:15:30,316 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6HV92s/build/debug/bin/postgresql.jar:/tmp/dist-test-task6HV92s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 04:15:30,558 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 04:15:30,559 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6HV92s/build/debug/bin/postgresql.jar:/tmp/dist-test-task6HV92s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41603/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6HV92s/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 04:15:31.516 UTC [12378] WARNING: there is no transaction in progress
2026-09-24 04:15:31.531 UTC [12378] WARNING: there is no transaction in progress
2026-09-24 04:15:31.547 UTC [12378] WARNING: there is no transaction in progress
2026-09-24 04:15:31.562 UTC [12378] WARNING: there is no transaction in progress
2026-09-24 04:15:31.579 UTC [12378] WARNING: there is no transaction in progress
2026-09-24 04:15:31.599 UTC [12378] WARNING: there is no transaction in progress
2026-09-24 04:15:31.653 UTC [12378] WARNING: there is no transaction in progress
2026-09-24 04:15:31.659 UTC [12378] WARNING: there is no transaction in progress
2026-09-24 04:15:31.673 UTC [12378] WARNING: there is no transaction in progress
2026-09-24 04:15:31.682 UTC [12378] WARNING: there is no transaction in progress
2026-09-24 04:15:31.692 UTC [12378] WARNING: there is no transaction in progress
2026-09-24 04:15:31.706 UTC [12378] WARNING: there is no transaction in progress
2026-09-24 04:15:31.716 UTC [12378] WARNING: there is no transaction in progress
2026-09-24 04:15:31.722 UTC [12378] WARNING: there is no transaction in progress
2026-09-24 04:15:31.729 UTC [12378] WARNING: there is no transaction in progress
2026-09-24 04:15:31.735 UTC [12378] WARNING: there is no transaction in progress
2026-09-24 04:15:32,366 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 04:15:32,366 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6HV92s/build/debug/bin/postgresql.jar:/tmp/dist-test-task6HV92s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 04:15:32,598 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 04:15:32,599 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6HV92s/build/debug/bin/postgresql.jar:/tmp/dist-test-task6HV92s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 04:15:32,845 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 04:15:32,845 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6HV92s/build/debug/bin/postgresql.jar:/tmp/dist-test-task6HV92s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 04:15:33,096 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 04:15:33,096 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6HV92s/build/debug/bin/postgresql.jar:/tmp/dist-test-task6HV92s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 04:15:33,323 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6HV92s/build/debug/bin/postgresql.jar:/tmp/dist-test-task6HV92s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 04:15:33,555 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6HV92s/build/debug/bin/postgresql.jar:/tmp/dist-test-task6HV92s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-p401.c.gcp-upstream.internal';"
2026-09-24 04:15:33,798 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 04:15:33,798 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6HV92s/build/debug/bin/postgresql.jar:/tmp/dist-test-task6HV92s/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 04:15:34,049 [I] DB_PATCHES have already been applied
I20260924 04:15:34.054258 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ranger-admin:/tmp/dist-test-task6HV92s/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task6HV92s/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task6HV92s/build/debug/bin/postgresql.jar:/tmp/dist-test-task6HV92s/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 04:15:34.054320 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260924 04:15:34.057480 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:39351
Sep 24, 2026 4:15:34 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 4:15:34 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 4:15:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 4:15:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 4:15:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 4:15:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 4:15:34 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 04:15:35.027973 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.103s user 0.000s sys 0.009s
Sep 24, 2026 4:15:55 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 4:15:55 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 24, 2026 4:15:55 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 4:15:55 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 4:15:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 4:15:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 4:15:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 4:15:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 4:15:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 4:15:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 4:15:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 4:15:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 4:15:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 4:15:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 4:15:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 4:15:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 4:15:55 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 4:15:56 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 04:15:56.978389 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 04:15:57 dist-test-slave-p401 krb5kdc[12165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223357, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 04:15:57.054322 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 04:15:57.062633 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 04:15:57.900220 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 04:15:57.900296 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ranger-kms
I20260924 04:15:57.900311 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:41603
2026-09-24 04:15:58,052 [I] DB FLAVOR :POSTGRES
2026-09-24 04:15:58,053 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 04:15:58,053 [I] Checking connection
2026-09-24 04:15:58,053 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6HV92s/build/debug/bin/postgresql.jar:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41603/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 04:15:58,284 [I] connection success
2026-09-24 04:15:58,285 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 04:15:58,285 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 04:15:58,285 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6HV92s/build/debug/bin/postgresql.jar:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41603/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 04:15:58,507 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 04:15:58,507 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 04:15:58,507 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 04:15:58,508 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6HV92s/build/debug/bin/postgresql.jar:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41603/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 04:15:58,777 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 04:15:59.191998 392 mini_ranger_kms.cc:310] Created kms service
I20260924 04:15:59.375851 392 mini_ranger_kms.cc:326] Created kudu user
I20260924 04:15:59.423699 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 04:15:59 dist-test-slave-p401 krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223357, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:15:59.803561 392 mini_ranger_kms.cc:384] Added ranger policy
I20260924 04:15:59.803709 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ranger-kms:/tmp/dist-test-task6HV92s/build/debug/bin/postgresql.jar:/tmp/dist-test-task6HV92s/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task6HV92s/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 04:15:59.803730 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260924 04:15:59.807147 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12558
I20260924 04:15:59.807219 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:43839
Sep 24, 2026 4:16:00 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 4:16:00 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 4:16:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 4:16:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 4:16:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 4:16:01 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 04:16:01.798585 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.744s user 0.010s sys 0.126s
I20260924 04:16:01.798707 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 04:16:01.798768 392 mini_ranger_kms.cc:401] 127.0.98.20:43839/kms/v1/keys
Sep 24 04:16:01 dist-test-slave-p401 krb5kdc[12165](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790223361, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 04:16:02 dist-test-slave-p401 krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223361, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 04:16:02 dist-test-slave-p401 krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223361, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 04:16:04 dist-test-slave-p401 krb5kdc[12165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:16:04.832414 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39963
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39963
--ranger_config_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:43839/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:16:04.941044 12609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:16:04.941288 12609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:16:04.941327 12609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:16:04.945363 12609 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:16:04.945420 12609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:16:04.945441 12609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:16:04.945459 12609 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:16:04.945488 12609 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:16:04.950912 12609 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:43839/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39963
--ranger_config_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:16:04.951944 12609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:16:04.952899 12609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:16:04.961521 12614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:16:04.961544 12615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:16:04.961598 12609 server_base.cc:1034] running on GCE node
W20260924 04:16:04.961544 12617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:16:04.962299 12609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:16:04.963383 12609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:16:04.964570 12609 hybrid_clock.cc:648] HybridClock initialized: now 1790223364964552 us; error 31 us; skew 500 ppm
Sep 24 04:16:04 dist-test-slave-p401 krb5kdc[12165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223364, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:16:04.967332 12609 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:16:04.968214 12609 webserver.cc:460] Webserver started at http://127.0.98.62:45575/ using document root <none> and password file <none>
I20260924 04:16:04.968724 12609 fs_manager.cc:359] Metadata directory not provided
I20260924 04:16:04.968780 12609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:16:04.968963 12609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 04:16:04 dist-test-slave-p401 krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223364, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:16:05.015704 12609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/master-0/data/instance:
uuid: "3bb47c28dd6b41769a35cef4cf70f3f7"
format_stamp: "Formatted at 2026-09-24 04:16:05 on dist-test-slave-p401"
server_key: "66e9b14bd599af559a2cf474e59a72a6"
server_key_iv: "1008cec286f83eebf985af122a985832"
server_key_version: "kuduclusterkey@0"
I20260924 04:16:05.016315 12609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/master-0/wal/instance:
uuid: "3bb47c28dd6b41769a35cef4cf70f3f7"
format_stamp: "Formatted at 2026-09-24 04:16:05 on dist-test-slave-p401"
server_key: "66e9b14bd599af559a2cf474e59a72a6"
server_key_iv: "1008cec286f83eebf985af122a985832"
server_key_version: "kuduclusterkey@0"
I20260924 04:16:05.020210 12609 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 04:16:05.039915 12624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:16:05.041473 12609 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:16:05.041610 12609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/master-0/wal
uuid: "3bb47c28dd6b41769a35cef4cf70f3f7"
format_stamp: "Formatted at 2026-09-24 04:16:05 on dist-test-slave-p401"
server_key: "66e9b14bd599af559a2cf474e59a72a6"
server_key_iv: "1008cec286f83eebf985af122a985832"
server_key_version: "kuduclusterkey@0"
I20260924 04:16:05.041752 12609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:16:05.054414 12609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:16:05.057716 12609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:16:05.057896 12609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:16:05.065289 12609 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39963
I20260924 04:16:05.065296 12676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39963 every 8 connection(s)
I20260924 04:16:05.066223 12609 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:16:05.068943 12677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:16:05.069111 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 12609
I20260924 04:16:05.069258 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/master-0/wal/instance
I20260924 04:16:05.069648 392 external_mini_cluster.cc:1428] Setting key 4cc39b61ffb3857fb006de5ecfb0588c
I20260924 04:16:05.076102 12677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bb47c28dd6b41769a35cef4cf70f3f7: Bootstrap starting.
I20260924 04:16:05.078588 12677 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3bb47c28dd6b41769a35cef4cf70f3f7: Neither blocks nor log segments found. Creating new log.
I20260924 04:16:05.079344 12677 log.cc:824] T 00000000000000000000000000000000 P 3bb47c28dd6b41769a35cef4cf70f3f7: Log is configured to *not* fsync() on all Append() calls
Sep 24 04:16:05 dist-test-slave-p401 krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:16:05.081645 12677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bb47c28dd6b41769a35cef4cf70f3f7: No bootstrap required, opened a new log
I20260924 04:16:05.085695 12680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:05.071222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34932 (local address 127.0.98.62:39963)
0924 04:16:05.071719 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:05.071747 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:05.071792 (+ 45us) server_negotiation.cc:407] Connection header received
0924 04:16:05.072502 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:05.072525 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:05.072959 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:05.073313 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:16:05.074265 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:05.075748 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:05.077046 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:05.077714 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:05.077880 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:05.080885 (+ 3005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:16:05.080890 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:16:05.080904 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:16:05.080929 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:16:05.083022 (+ 2093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:05.083506 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:05.083511 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:05.083516 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:05.083617 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:05.083869 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:05.083873 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:05.083874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:05.084244 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:16:05.084427 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:05.084687 (+ 260us) server_negotiation.cc:299] Negotiation successful
0924 04:16:05.084980 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":159,"threads_started":1}
I20260924 04:16:05.085994 12677 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bb47c28dd6b41769a35cef4cf70f3f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bb47c28dd6b41769a35cef4cf70f3f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39963 } }
I20260924 04:16:05.086318 12677 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3bb47c28dd6b41769a35cef4cf70f3f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:16:05.086447 12677 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bb47c28dd6b41769a35cef4cf70f3f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bb47c28dd6b41769a35cef4cf70f3f7, State: Initialized, Role: FOLLOWER
I20260924 04:16:05.086944 12677 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bb47c28dd6b41769a35cef4cf70f3f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bb47c28dd6b41769a35cef4cf70f3f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39963 } }
I20260924 04:16:05.087124 12677 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3bb47c28dd6b41769a35cef4cf70f3f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:16:05.087270 12677 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3bb47c28dd6b41769a35cef4cf70f3f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:16:05.087388 12677 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bb47c28dd6b41769a35cef4cf70f3f7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:16:05.088626 12677 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3bb47c28dd6b41769a35cef4cf70f3f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bb47c28dd6b41769a35cef4cf70f3f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39963 } }
I20260924 04:16:05.089001 12677 leader_election.cc:302] T 00000000000000000000000000000000 P 3bb47c28dd6b41769a35cef4cf70f3f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3bb47c28dd6b41769a35cef4cf70f3f7; no voters:
I20260924 04:16:05.089376 12677 leader_election.cc:288] T 00000000000000000000000000000000 P 3bb47c28dd6b41769a35cef4cf70f3f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:16:05.089476 12682 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3bb47c28dd6b41769a35cef4cf70f3f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:16:05.089681 12682 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3bb47c28dd6b41769a35cef4cf70f3f7 [term 1 LEADER]: Becoming Leader. State: Replica: 3bb47c28dd6b41769a35cef4cf70f3f7, State: Running, Role: LEADER
I20260924 04:16:05.090024 12682 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3bb47c28dd6b41769a35cef4cf70f3f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bb47c28dd6b41769a35cef4cf70f3f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39963 } }
I20260924 04:16:05.090736 12677 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bb47c28dd6b41769a35cef4cf70f3f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:16:05.091596 12684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bb47c28dd6b41769a35cef4cf70f3f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bb47c28dd6b41769a35cef4cf70f3f7. Latest consensus state: current_term: 1 leader_uuid: "3bb47c28dd6b41769a35cef4cf70f3f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bb47c28dd6b41769a35cef4cf70f3f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39963 } } }
I20260924 04:16:05.091723 12684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bb47c28dd6b41769a35cef4cf70f3f7 [sys.catalog]: This master's current role is: LEADER
I20260924 04:16:05.092098 12683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bb47c28dd6b41769a35cef4cf70f3f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3bb47c28dd6b41769a35cef4cf70f3f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bb47c28dd6b41769a35cef4cf70f3f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39963 } } }
I20260924 04:16:05.092185 12683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bb47c28dd6b41769a35cef4cf70f3f7 [sys.catalog]: This master's current role is: LEADER
I20260924 04:16:05.093230 12677 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 04:16:05.094004 12691 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:16:05.098995 12691 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:16:05.106604 12691 catalog_manager.cc:1347] Generated new cluster ID: 660a8a62d5fa47b3a7252bab1e053299
I20260924 04:16:05.106671 12691 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:16:05.124681 12691 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:16:05.126040 12691 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:16:05.144336 12691 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3bb47c28dd6b41769a35cef4cf70f3f7: Generated new TSK 0
I20260924 04:16:05.145062 12691 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:16:05.814911 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:43839/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39963
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:16:05.959609 12722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:16:05.959862 12722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:16:05.959906 12722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:16:05.963940 12722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:16:05.963997 12722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:16:05.964058 12722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:16:05.969313 12722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:43839/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:16:05.970345 12722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:16:05.971472 12722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:16:05.982100 12729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:16:05.982100 12728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:16:05.982479 12731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:16:05.983417 12722 server_base.cc:1034] running on GCE node
I20260924 04:16:05.983820 12722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:16:05.984527 12722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:16:05.985724 12722 hybrid_clock.cc:648] HybridClock initialized: now 1790223365985698 us; error 37 us; skew 500 ppm
Sep 24 04:16:05 dist-test-slave-p401 krb5kdc[12165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223365, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:16:05.988994 12722 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:16:05.990056 12722 webserver.cc:460] Webserver started at http://127.0.98.1:43649/ using document root <none> and password file <none>
I20260924 04:16:05.990765 12722 fs_manager.cc:359] Metadata directory not provided
I20260924 04:16:05.990856 12722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:16:05.991148 12722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 04:16:05 dist-test-slave-p401 krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223365, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:16:06.007520 12722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-0/data/instance:
uuid: "ba934cbd2b56481cbc1743baffe26a49"
format_stamp: "Formatted at 2026-09-24 04:16:05 on dist-test-slave-p401"
server_key: "9941f504f7949d75f55fb262f654ec85"
server_key_iv: "362a0c1e39a9b16f351b2183120e946b"
server_key_version: "kuduclusterkey@0"
I20260924 04:16:06.008317 12722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ba934cbd2b56481cbc1743baffe26a49"
format_stamp: "Formatted at 2026-09-24 04:16:05 on dist-test-slave-p401"
server_key: "9941f504f7949d75f55fb262f654ec85"
server_key_iv: "362a0c1e39a9b16f351b2183120e946b"
server_key_version: "kuduclusterkey@0"
I20260924 04:16:06.018555 12722 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.004s sys 0.000s
I20260924 04:16:06.035642 12739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:16:06.037035 12722 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 04:16:06.037155 12722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-0/wal
uuid: "ba934cbd2b56481cbc1743baffe26a49"
format_stamp: "Formatted at 2026-09-24 04:16:05 on dist-test-slave-p401"
server_key: "9941f504f7949d75f55fb262f654ec85"
server_key_iv: "362a0c1e39a9b16f351b2183120e946b"
server_key_version: "kuduclusterkey@0"
I20260924 04:16:06.037279 12722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:16:06.077657 12722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:16:06.080914 12722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:16:06.081111 12722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:16:06.081818 12722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:16:06.082839 12722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:16:06.082901 12722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:16:06.082953 12722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:16:06.083029 12722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:16:06.093240 12722 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35115
I20260924 04:16:06.094417 12722 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:16:06.095106 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 12722
I20260924 04:16:06.095253 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:16:06.095558 392 external_mini_cluster.cc:1428] Setting key b36bdf2eddbeb75fdf759848dc7ec6af
I20260924 04:16:06.096752 12854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35115 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Sep 24 04:16:06 dist-test-slave-p401 krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223365, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:16:06.124722 12859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:06.101193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43915 (local address 127.0.98.62:39963)
0924 04:16:06.108488 (+ 7295us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:06.108496 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:06.108511 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:16:06.108554 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:06.108557 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:06.108603 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:06.108691 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:16:06.110458 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:06.111298 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:06.112712 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:06.113051 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:06.113154 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:06.119977 (+ 6823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:16:06.119981 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:16:06.119985 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:16:06.120018 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:16:06.121815 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:06.122524 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:06.122528 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:06.122530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:06.122629 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:06.123258 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:06.123262 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:06.123264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:06.123517 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:16:06.123677 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:06.124400 (+ 723us) server_negotiation.cc:299] Negotiation successful
0924 04:16:06.124534 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7211,"thread_start_us":86,"threads_started":1}
I20260924 04:16:06.125967 12858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:06.100116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39963 (local address 127.0.98.1:43915)
0924 04:16:06.106488 (+ 6372us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:06.106554 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0924 04:16:06.107660 (+ 1106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:16:06.109012 (+ 1352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:16:06.109023 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:16:06.109395 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:16:06.110122 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:06.110144 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:06.111454 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:06.111457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:06.112567 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:06.112579 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:06.113224 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:06.113228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:06.113419 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:06.114050 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:16:06.114074 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:16:06.119656 (+ 5582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:16:06.121990 (+ 2334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:06.121998 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:06.122020 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:06.122398 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:06.122878 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:06.122882 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:06.122884 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:06.123103 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:06.123683 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:16:06.123691 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:16:06.124083 (+ 392us) client_negotiation.cc:769] Sending connection context
0924 04:16:06.124325 (+ 242us) client_negotiation.cc:240] Negotiation successful
0924 04:16:06.124705 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6106,"thread_start_us":133,"threads_started":1}
I20260924 04:16:06.127434 12855 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39963
I20260924 04:16:06.127701 12855 heartbeater.cc:464] Registering TS with master...
I20260924 04:16:06.128311 12855 heartbeater.cc:511] Master 127.0.98.62:39963 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:16:06.130187 12641 ts_manager.cc:194] Registered new tserver with Master: ba934cbd2b56481cbc1743baffe26a49 (127.0.98.1:35115)
I20260924 04:16:06.130831 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:43839/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39963
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:16:06.131775 12641 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43915
Sep 24 04:16:06 dist-test-slave-p401 krb5kdc[12165](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790223366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 04:16:06.270496 12861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:16:06.270874 12861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:16:06.270960 12861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:16:06.277020 12861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:16:06.277096 12861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:16:06.277182 12861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:16:06.284948 12861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:43839/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:16:06.286410 12861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:16:06.287823 12861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:16:06.300101 12867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:16:06.300100 12870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:16:06.300213 12868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:16:06.300750 12861 server_base.cc:1034] running on GCE node
I20260924 04:16:06.301251 12861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:16:06.302145 12861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:16:06.303310 12861 hybrid_clock.cc:648] HybridClock initialized: now 1790223366303280 us; error 43 us; skew 500 ppm
Sep 24 04:16:06 dist-test-slave-p401 krb5kdc[12165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:16:06.307032 12861 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:16:06.308200 12861 webserver.cc:460] Webserver started at http://127.0.98.2:44479/ using document root <none> and password file <none>
I20260924 04:16:06.308899 12861 fs_manager.cc:359] Metadata directory not provided
I20260924 04:16:06.309015 12861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:16:06.309260 12861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 04:16:06 dist-test-slave-p401 krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:16:06.325459 12861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-1/data/instance:
uuid: "d61a304adedb490891eb590885498ca0"
format_stamp: "Formatted at 2026-09-24 04:16:06 on dist-test-slave-p401"
server_key: "ef1c20f13b4a3889558e20fd5cfa76c5"
server_key_iv: "bf3530d3978af6cf08705b03532df0d3"
server_key_version: "kuduclusterkey@0"
I20260924 04:16:06.326282 12861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d61a304adedb490891eb590885498ca0"
format_stamp: "Formatted at 2026-09-24 04:16:06 on dist-test-slave-p401"
server_key: "ef1c20f13b4a3889558e20fd5cfa76c5"
server_key_iv: "bf3530d3978af6cf08705b03532df0d3"
server_key_version: "kuduclusterkey@0"
I20260924 04:16:06.331635 12861 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 04:16:06.359479 12877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:16:06.361296 12861 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 04:16:06.361434 12861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-1/wal
uuid: "d61a304adedb490891eb590885498ca0"
format_stamp: "Formatted at 2026-09-24 04:16:06 on dist-test-slave-p401"
server_key: "ef1c20f13b4a3889558e20fd5cfa76c5"
server_key_iv: "bf3530d3978af6cf08705b03532df0d3"
server_key_version: "kuduclusterkey@0"
I20260924 04:16:06.361578 12861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:16:06.424065 12861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:16:06.427793 12861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:16:06.427983 12861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:16:06.428678 12861 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:16:06.429713 12861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:16:06.429774 12861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:16:06.429826 12861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:16:06.429899 12861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260924 04:16:06.440289 12861 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34937
I20260924 04:16:06.441462 12861 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:16:06.442309 12990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34937 every 8 connection(s)
I20260924 04:16:06.447117 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 12861
I20260924 04:16:06.447258 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:16:06.447540 392 external_mini_cluster.cc:1428] Setting key c5360adb116012a37fa40ad776d05cef
Sep 24 04:16:06 dist-test-slave-p401 krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:16:06.458665 12859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:06.443663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57083 (local address 127.0.98.62:39963)
0924 04:16:06.443799 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:06.443803 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:06.444987 (+ 1184us) server_negotiation.cc:407] Connection header received
0924 04:16:06.446390 (+ 1403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:06.446393 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:06.446450 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:06.446546 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:16:06.448181 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:06.448994 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:06.450804 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:06.451145 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:06.451248 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:06.454439 (+ 3191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:16:06.454443 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:16:06.454446 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:16:06.454475 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:16:06.456147 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:06.456778 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:06.456782 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:06.456784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:06.456877 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:06.457295 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:06.457299 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:06.457301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:06.457553 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:16:06.457652 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:06.458360 (+ 708us) server_negotiation.cc:299] Negotiation successful
0924 04:16:06.458496 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 04:16:06.459947 12993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:06.444221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39963 (local address 127.0.98.2:57083)
0924 04:16:06.444817 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:06.444878 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 04:16:06.446085 (+ 1207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:16:06.446767 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:16:06.446778 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:16:06.447179 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:16:06.447988 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:06.448010 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:06.449168 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:06.449171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:06.450652 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:06.450664 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:06.451502 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:06.451507 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:06.451730 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:06.452433 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:16:06.452454 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:16:06.454246 (+ 1792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:16:06.456290 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:06.456296 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:06.456314 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:06.456646 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:06.456984 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:06.456988 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:06.456990 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:06.457185 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:06.457666 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:16:06.457672 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:16:06.458077 (+ 405us) client_negotiation.cc:769] Sending connection context
0924 04:16:06.458320 (+ 243us) client_negotiation.cc:240] Negotiation successful
0924 04:16:06.458676 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":141,"threads_started":1}
I20260924 04:16:06.461201 12991 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39963
I20260924 04:16:06.461457 12991 heartbeater.cc:464] Registering TS with master...
I20260924 04:16:06.462031 12991 heartbeater.cc:511] Master 127.0.98.62:39963 requested a full tablet report, sending...
I20260924 04:16:06.463160 12641 ts_manager.cc:194] Registered new tserver with Master: d61a304adedb490891eb590885498ca0 (127.0.98.2:34937)
I20260924 04:16:06.463917 12641 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57083
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:16:06.494530 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:43839/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39963
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:16:06.660972 12996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:16:06.661736 12996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:16:06.661875 12996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:16:06.668078 12996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:16:06.668211 12996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:16:06.668335 12996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:16:06.676043 12996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:43839/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:16:06.677626 12996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:16:06.679001 12996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:16:06.691699 13002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:16:06.692445 13001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:16:06.692749 13004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:16:06.694010 12996 server_base.cc:1034] running on GCE node
I20260924 04:16:06.694646 12996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:16:06.695505 12996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:16:06.696657 12996 hybrid_clock.cc:648] HybridClock initialized: now 1790223366696590 us; error 83 us; skew 500 ppm
Sep 24 04:16:06 dist-test-slave-p401 krb5kdc[12165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:16:06.700337 12996 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:16:06.701448 12996 webserver.cc:460] Webserver started at http://127.0.98.3:39037/ using document root <none> and password file <none>
I20260924 04:16:06.702152 12996 fs_manager.cc:359] Metadata directory not provided
I20260924 04:16:06.702217 12996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:16:06.702430 12996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 04:16:06 dist-test-slave-p401 krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 04:16:06 dist-test-slave-p401 krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:16:06.726817 12996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-2/data/instance:
uuid: "9555a575bb054c34b1ebfae7e91303a4"
format_stamp: "Formatted at 2026-09-24 04:16:06 on dist-test-slave-p401"
server_key: "0a166328244462f19bc34b0b0ca78a0d"
server_key_iv: "b8236a530c97ce26f7c235d95c87c899"
server_key_version: "kuduclusterkey@0"
I20260924 04:16:06.731559 12996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9555a575bb054c34b1ebfae7e91303a4"
format_stamp: "Formatted at 2026-09-24 04:16:06 on dist-test-slave-p401"
server_key: "0a166328244462f19bc34b0b0ca78a0d"
server_key_iv: "b8236a530c97ce26f7c235d95c87c899"
server_key_version: "kuduclusterkey@0"
I20260924 04:16:06.737149 12996 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 04:16:06.754437 13011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:16:06.756137 12996 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 04:16:06.756330 12996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-2/wal
uuid: "9555a575bb054c34b1ebfae7e91303a4"
format_stamp: "Formatted at 2026-09-24 04:16:06 on dist-test-slave-p401"
server_key: "0a166328244462f19bc34b0b0ca78a0d"
server_key_iv: "b8236a530c97ce26f7c235d95c87c899"
server_key_version: "kuduclusterkey@0"
I20260924 04:16:06.756518 12996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:16:06.776927 12996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:16:06.780895 12996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:16:06.781143 12996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:16:06.781939 12996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:16:06.782994 12996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:16:06.783092 12996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:16:06.783181 12996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:16:06.783277 12996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:16:06.795851 12996 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43629
I20260924 04:16:06.797058 12996 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:16:06.800709 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 12996
I20260924 04:16:06.800827 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:16:06.801111 392 external_mini_cluster.cc:1428] Setting key 203c49020e6e48dbb1e96121268da027
I20260924 04:16:06.803583 13124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43629 every 8 connection(s)
Sep 24 04:16:06 dist-test-slave-p401 krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:16:06.837756 12859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:06.818795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55467 (local address 127.0.98.62:39963)
0924 04:16:06.818916 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:06.818921 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:06.819751 (+ 830us) server_negotiation.cc:407] Connection header received
0924 04:16:06.820998 (+ 1247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:06.821002 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:06.821053 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:06.821108 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:16:06.822772 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:06.823622 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:06.825317 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:06.825647 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:06.825723 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:06.833472 (+ 7749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:16:06.833476 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:16:06.833479 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:16:06.833509 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:16:06.835199 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:06.835859 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:06.835863 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:06.835865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:06.835960 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:06.836406 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:06.836410 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:06.836411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:06.836639 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:16:06.836709 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:06.837457 (+ 748us) server_negotiation.cc:299] Negotiation successful
0924 04:16:06.837585 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 04:16:06.839694 13127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:06.817688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39963 (local address 127.0.98.3:55467)
0924 04:16:06.819594 (+ 1906us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:06.819655 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 04:16:06.820745 (+ 1090us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:16:06.821526 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:16:06.821535 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:16:06.821902 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:16:06.822603 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:06.822626 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:06.823771 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:06.823774 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:06.825169 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:06.825180 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:06.825917 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:06.825921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:06.826108 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:06.831407 (+ 5299us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:16:06.831447 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:16:06.833246 (+ 1799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:16:06.835366 (+ 2120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:06.835372 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:06.835391 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:06.835735 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:06.836072 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:06.836075 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:06.836077 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:06.836294 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:06.836790 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:16:06.836797 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:16:06.837161 (+ 364us) client_negotiation.cc:769] Sending connection context
0924 04:16:06.838228 (+ 1067us) client_negotiation.cc:240] Negotiation successful
0924 04:16:06.838592 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1676,"thread_start_us":159,"threads_started":1}
I20260924 04:16:06.841130 13125 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39963
I20260924 04:16:06.841369 13125 heartbeater.cc:464] Registering TS with master...
I20260924 04:16:06.841936 13125 heartbeater.cc:511] Master 127.0.98.62:39963 requested a full tablet report, sending...
I20260924 04:16:06.843062 12641 ts_manager.cc:194] Registered new tserver with Master: 9555a575bb054c34b1ebfae7e91303a4 (127.0.98.3:43629)
I20260924 04:16:06.843843 12641 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55467
I20260924 04:16:06.850178 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:16:06.850981 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 12722
I20260924 04:16:06.857766 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 12861
I20260924 04:16:06.865193 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 12996
I20260924 04:16:06.871803 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 12609
I20260924 04:16:06.878756 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39963
--webserver_interface=127.0.98.62
--webserver_port=45575
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39963
--ranger_config_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:43839/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:43839/kms with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:16:07.037873 13128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:16:07.038252 13128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:16:07.038324 13128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:16:07.044416 13128 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:16:07.044495 13128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:16:07.044521 13128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:16:07.044550 13128 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:16:07.044576 13128 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:16:07.052000 13128 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:43839/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39963
--ranger_config_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=45575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:16:07.052984 13128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:16:07.053936 13128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:16:07.062026 13136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:16:07.062033 13133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:16:07.062112 13128 server_base.cc:1034] running on GCE node
W20260924 04:16:07.062089 13134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:16:07.062765 13128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:16:07.063824 13128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:16:07.064975 13128 hybrid_clock.cc:648] HybridClock initialized: now 1790223367064956 us; error 36 us; skew 500 ppm
Sep 24 04:16:07 dist-test-slave-p401 krb5kdc[12165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223367, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:16:07.067595 13128 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:16:07.068395 13128 webserver.cc:460] Webserver started at http://127.0.98.62:45575/ using document root <none> and password file <none>
I20260924 04:16:07.068874 13128 fs_manager.cc:359] Metadata directory not provided
I20260924 04:16:07.068940 13128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 04:16:07 dist-test-slave-p401 krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223367, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:16:07.141460 13128 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 04:16:07.144165 13144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:16:07.145696 13128 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:16:07.145817 13128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/master-0/wal
uuid: "3bb47c28dd6b41769a35cef4cf70f3f7"
format_stamp: "Formatted at 2026-09-24 04:16:05 on dist-test-slave-p401"
server_key: "66e9b14bd599af559a2cf474e59a72a6"
server_key_iv: "1008cec286f83eebf985af122a985832"
server_key_version: "kuduclusterkey@0"
I20260924 04:16:07.146365 13128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:16:07.165346 13128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:16:07.168187 13128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:16:07.168365 13128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:16:07.175395 13128 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39963
I20260924 04:16:07.175433 13196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39963 every 8 connection(s)
I20260924 04:16:07.176486 13128 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:16:07.179167 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 13128
I20260924 04:16:07.179664 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:43839/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:35115
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:39963
--webserver_port=43649
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:43839/kms with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:16:07.181171 13197 sys_catalog.cc:263] Verifying existing consensus state
I20260924 04:16:07.182837 13197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bb47c28dd6b41769a35cef4cf70f3f7: Bootstrap starting.
I20260924 04:16:07.198935 13197 log.cc:824] T 00000000000000000000000000000000 P 3bb47c28dd6b41769a35cef4cf70f3f7: Log is configured to *not* fsync() on all Append() calls
I20260924 04:16:07.205381 13197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bb47c28dd6b41769a35cef4cf70f3f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:16:07.205754 13197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bb47c28dd6b41769a35cef4cf70f3f7: Bootstrap complete.
I20260924 04:16:07.209681 13197 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bb47c28dd6b41769a35cef4cf70f3f7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bb47c28dd6b41769a35cef4cf70f3f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39963 } }
I20260924 04:16:07.210074 13197 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bb47c28dd6b41769a35cef4cf70f3f7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bb47c28dd6b41769a35cef4cf70f3f7, State: Initialized, Role: FOLLOWER
I20260924 04:16:07.210565 13197 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bb47c28dd6b41769a35cef4cf70f3f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bb47c28dd6b41769a35cef4cf70f3f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39963 } }
I20260924 04:16:07.210676 13197 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3bb47c28dd6b41769a35cef4cf70f3f7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:16:07.210768 13197 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3bb47c28dd6b41769a35cef4cf70f3f7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:16:07.210865 13197 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bb47c28dd6b41769a35cef4cf70f3f7 [term 1 FOLLOWER]: Advancing to term 2
I20260924 04:16:07.212932 13197 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3bb47c28dd6b41769a35cef4cf70f3f7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bb47c28dd6b41769a35cef4cf70f3f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39963 } }
I20260924 04:16:07.213294 13197 leader_election.cc:302] T 00000000000000000000000000000000 P 3bb47c28dd6b41769a35cef4cf70f3f7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3bb47c28dd6b41769a35cef4cf70f3f7; no voters:
I20260924 04:16:07.213625 13197 leader_election.cc:288] T 00000000000000000000000000000000 P 3bb47c28dd6b41769a35cef4cf70f3f7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 04:16:07.213709 13201 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3bb47c28dd6b41769a35cef4cf70f3f7 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 04:16:07.213939 13201 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3bb47c28dd6b41769a35cef4cf70f3f7 [term 2 LEADER]: Becoming Leader. State: Replica: 3bb47c28dd6b41769a35cef4cf70f3f7, State: Running, Role: LEADER
I20260924 04:16:07.214299 13201 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3bb47c28dd6b41769a35cef4cf70f3f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bb47c28dd6b41769a35cef4cf70f3f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39963 } }
I20260924 04:16:07.214852 13197 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bb47c28dd6b41769a35cef4cf70f3f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:16:07.215324 13203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bb47c28dd6b41769a35cef4cf70f3f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bb47c28dd6b41769a35cef4cf70f3f7. Latest consensus state: current_term: 2 leader_uuid: "3bb47c28dd6b41769a35cef4cf70f3f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bb47c28dd6b41769a35cef4cf70f3f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39963 } } }
I20260924 04:16:07.215448 13203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bb47c28dd6b41769a35cef4cf70f3f7 [sys.catalog]: This master's current role is: LEADER
I20260924 04:16:07.215647 13202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bb47c28dd6b41769a35cef4cf70f3f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3bb47c28dd6b41769a35cef4cf70f3f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bb47c28dd6b41769a35cef4cf70f3f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39963 } } }
I20260924 04:16:07.215762 13202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bb47c28dd6b41769a35cef4cf70f3f7 [sys.catalog]: This master's current role is: LEADER
I20260924 04:16:07.217025 13205 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:16:07.217362 13197 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 04:16:07.222995 13205 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:16:07.223932 13205 catalog_manager.cc:1259] Loaded cluster ID: 660a8a62d5fa47b3a7252bab1e053299
I20260924 04:16:07.224032 13205 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:16:07.226786 13205 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:16:07.227545 13205 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3bb47c28dd6b41769a35cef4cf70f3f7: Loaded TSK: 0
I20260924 04:16:07.228816 13205 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 04:16:07.314414 13199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:16:07.314656 13199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:16:07.314713 13199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:16:07.319555 13199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:16:07.319669 13199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:16:07.319783 13199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:16:07.326674 13199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:43839/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:35115
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43649
--tserver_master_addrs=127.0.98.62:39963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:16:07.327962 13199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:16:07.329046 13199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:16:07.338647 13234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:16:07.338604 13237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:16:07.339093 13235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:16:07.339378 13199 server_base.cc:1034] running on GCE node
I20260924 04:16:07.339923 13199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:16:07.340660 13199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:16:07.341822 13199 hybrid_clock.cc:648] HybridClock initialized: now 1790223367341810 us; error 25 us; skew 500 ppm
Sep 24 04:16:07 dist-test-slave-p401 krb5kdc[12165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223367, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:16:07.344497 13199 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:16:07.345515 13199 webserver.cc:460] Webserver started at http://127.0.98.1:43649/ using document root <none> and password file <none>
I20260924 04:16:07.346197 13199 fs_manager.cc:359] Metadata directory not provided
I20260924 04:16:07.346354 13199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 04:16:07 dist-test-slave-p401 krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223367, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:16:07.386222 13199 fs_manager.cc:709] Time spent opening directory manager: real 0.010s user 0.003s sys 0.000s
I20260924 04:16:07.389778 13245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:16:07.391489 13199 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 04:16:07.391628 13199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-0/wal
uuid: "ba934cbd2b56481cbc1743baffe26a49"
format_stamp: "Formatted at 2026-09-24 04:16:05 on dist-test-slave-p401"
server_key: "9941f504f7949d75f55fb262f654ec85"
server_key_iv: "362a0c1e39a9b16f351b2183120e946b"
server_key_version: "kuduclusterkey@0"
I20260924 04:16:07.392390 13199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:16:07.406462 13199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:16:07.409966 13199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:16:07.410189 13199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:16:07.410806 13199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:16:07.411760 13199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:16:07.411852 13199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:16:07.411906 13199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:16:07.411948 13199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260924 04:16:07.422159 13199 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35115
I20260924 04:16:07.422308 13358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35115 every 8 connection(s)
I20260924 04:16:07.423573 13199 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:16:07.433386 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 13199
I20260924 04:16:07.433892 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:43839/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34937
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:39963
--webserver_port=44479
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:43839/kms with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 04:16:07 dist-test-slave-p401 krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223367, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:16:07.454510 13361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:07.426196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39963 (local address 127.0.98.1:38367)
0924 04:16:07.426786 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:07.426843 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 04:16:07.427908 (+ 1065us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:16:07.429824 (+ 1916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:16:07.429833 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:16:07.430180 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:16:07.430872 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:07.430893 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:07.435672 (+ 4779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:07.435676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:07.437138 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:07.437151 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:07.438222 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:07.438227 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:07.438428 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:07.439111 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:16:07.439132 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:16:07.440843 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:16:07.450606 (+ 9763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:07.450614 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:07.450709 (+ 95us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:07.451119 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:07.451616 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:07.451619 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:07.451621 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:07.451823 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:07.452633 (+ 810us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:16:07.452640 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:16:07.452953 (+ 313us) client_negotiation.cc:769] Sending connection context
0924 04:16:07.453146 (+ 193us) client_negotiation.cc:240] Negotiation successful
0924 04:16:07.453492 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":151,"threads_started":1}
I20260924 04:16:07.456444 13362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:07.427718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38367 (local address 127.0.98.62:39963)
0924 04:16:07.428409 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:07.428431 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:07.428477 (+ 46us) server_negotiation.cc:407] Connection header received
0924 04:16:07.428817 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:07.428882 (+ 65us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:07.429321 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:07.431118 (+ 1797us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:16:07.431486 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:07.433067 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:07.437299 (+ 4232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:07.438016 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:07.438183 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:07.443387 (+ 5204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:16:07.443395 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:16:07.443414 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:16:07.443450 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:16:07.450422 (+ 6972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:07.451309 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:07.451316 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:07.451323 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:07.451501 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:07.451946 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:07.451949 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:07.451951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:07.452452 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:16:07.452569 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:07.455045 (+ 2476us) server_negotiation.cc:299] Negotiation successful
0924 04:16:07.455502 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":120,"threads_started":1}
I20260924 04:16:07.459205 13359 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39963
I20260924 04:16:07.459482 13359 heartbeater.cc:464] Registering TS with master...
I20260924 04:16:07.460017 13359 heartbeater.cc:511] Master 127.0.98.62:39963 requested a full tablet report, sending...
W20260924 04:16:07.462215 13359 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39963 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 04:16:07.462353 13359 heartbeater.cc:464] Registering TS with master...
I20260924 04:16:07.462531 13359 heartbeater.cc:511] Master 127.0.98.62:39963 requested a full tablet report, sending...
I20260924 04:16:07.463573 13359 heartbeater.cc:464] Registering TS with master...
I20260924 04:16:07.463752 13359 heartbeater.cc:511] Master 127.0.98.62:39963 requested a full tablet report, sending...
W20260924 04:16:07.464561 13359 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 04:16:07.575341 13363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:16:07.575616 13363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:16:07.575678 13363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:16:07.580402 13363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:16:07.580475 13363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:16:07.580559 13363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:16:07.586279 13363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:43839/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34937
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=44479
--tserver_master_addrs=127.0.98.62:39963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:16:07.587450 13363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:16:07.588521 13363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:16:07.598160 13368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:16:07.598178 13369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:16:07.598641 13371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:16:07.598968 13363 server_base.cc:1034] running on GCE node
I20260924 04:16:07.599395 13363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:16:07.600009 13363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:16:07.603312 13363 hybrid_clock.cc:648] HybridClock initialized: now 1790223367603279 us; error 60 us; skew 500 ppm
Sep 24 04:16:07 dist-test-slave-p401 krb5kdc[12165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223367, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:16:07.606005 13363 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:16:07.606863 13363 webserver.cc:460] Webserver started at http://127.0.98.2:44479/ using document root <none> and password file <none>
I20260924 04:16:07.607450 13363 fs_manager.cc:359] Metadata directory not provided
I20260924 04:16:07.607533 13363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 04:16:07 dist-test-slave-p401 krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223367, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:16:07.627477 13363 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 04:16:07.639714 13379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:16:07.641656 13363 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.001s sys 0.000s
I20260924 04:16:07.641881 13363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-1/wal
uuid: "d61a304adedb490891eb590885498ca0"
format_stamp: "Formatted at 2026-09-24 04:16:06 on dist-test-slave-p401"
server_key: "ef1c20f13b4a3889558e20fd5cfa76c5"
server_key_iv: "bf3530d3978af6cf08705b03532df0d3"
server_key_version: "kuduclusterkey@0"
I20260924 04:16:07.642730 13363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:16:07.662890 13363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:16:07.666263 13363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:16:07.666508 13363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:16:07.667289 13363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:16:07.668391 13363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:16:07.668442 13363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:16:07.668493 13363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:16:07.668512 13363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:16:07.678910 13363 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34937
I20260924 04:16:07.680294 13363 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:16:07.684857 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 13363
I20260924 04:16:07.685298 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:43839/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:43629
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:39963
--webserver_port=39037
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:43839/kms with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:16:07.687827 13492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34937 every 8 connection(s)
Sep 24 04:16:07 dist-test-slave-p401 krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223367, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:16:07.700891 13362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:07.682318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42749 (local address 127.0.98.62:39963)
0924 04:16:07.682449 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:07.682453 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:07.683590 (+ 1137us) server_negotiation.cc:407] Connection header received
0924 04:16:07.687178 (+ 3588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:07.687182 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:07.687286 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:07.687407 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:16:07.688830 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:07.689721 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:07.691533 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:07.691903 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:07.691988 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:07.696453 (+ 4465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:16:07.696458 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:16:07.696461 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:16:07.696494 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:16:07.698268 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:07.698925 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:07.698929 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:07.698931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:07.699023 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:07.699505 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:07.699508 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:07.699510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:07.699748 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:16:07.699834 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:07.700582 (+ 748us) server_negotiation.cc:299] Negotiation successful
0924 04:16:07.700718 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 04:16:07.702800 13495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:07.682854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39963 (local address 127.0.98.2:42749)
0924 04:16:07.683428 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:07.683485 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 04:16:07.684571 (+ 1086us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:16:07.687643 (+ 3072us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:16:07.687653 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:16:07.688018 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:16:07.688660 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:07.688679 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:07.689870 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:07.689873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:07.691366 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:07.691378 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:07.692205 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:07.692209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:07.692418 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:07.693041 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:16:07.693064 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:16:07.696230 (+ 3166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:16:07.698405 (+ 2175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:07.698411 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:07.698429 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:07.698795 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:07.699133 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:07.699136 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:07.699138 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:07.699380 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:07.699902 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:16:07.699909 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:16:07.700278 (+ 369us) client_negotiation.cc:769] Sending connection context
0924 04:16:07.701316 (+ 1038us) client_negotiation.cc:240] Negotiation successful
0924 04:16:07.701692 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":139,"threads_started":1}
I20260924 04:16:07.704494 13493 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39963
I20260924 04:16:07.704751 13493 heartbeater.cc:464] Registering TS with master...
I20260924 04:16:07.705214 13493 heartbeater.cc:511] Master 127.0.98.62:39963 requested a full tablet report, sending...
W20260924 04:16:07.706692 13493 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39963 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 04:16:07.706820 13493 heartbeater.cc:464] Registering TS with master...
I20260924 04:16:07.706988 13493 heartbeater.cc:511] Master 127.0.98.62:39963 requested a full tablet report, sending...
I20260924 04:16:07.707768 13493 heartbeater.cc:464] Registering TS with master...
I20260924 04:16:07.707933 13493 heartbeater.cc:511] Master 127.0.98.62:39963 requested a full tablet report, sending...
W20260924 04:16:07.708636 13493 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 04:16:07.825521 13496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:16:07.825909 13496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:16:07.825994 13496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:16:07.830154 13496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:16:07.830209 13496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:16:07.830265 13496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:16:07.835877 13496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:43839/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:43629
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39037
--tserver_master_addrs=127.0.98.62:39963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:16:07.837110 13496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:16:07.838326 13496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:16:07.851750 13502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:16:07.851812 13505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:16:07.852218 13496 server_base.cc:1034] running on GCE node
W20260924 04:16:07.855693 13503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:16:07.856113 13496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:16:07.856834 13496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:16:07.858001 13496 hybrid_clock.cc:648] HybridClock initialized: now 1790223367857976 us; error 38 us; skew 500 ppm
Sep 24 04:16:07 dist-test-slave-p401 krb5kdc[12165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223367, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:16:07.861442 13496 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:16:07.862598 13496 webserver.cc:460] Webserver started at http://127.0.98.3:39037/ using document root <none> and password file <none>
I20260924 04:16:07.863399 13496 fs_manager.cc:359] Metadata directory not provided
I20260924 04:16:07.863554 13496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 04:16:07 dist-test-slave-p401 krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223367, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:16:07.883862 13496 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:16:07.886942 13513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:16:07.888623 13496 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:16:07.888794 13496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-2/wal
uuid: "9555a575bb054c34b1ebfae7e91303a4"
format_stamp: "Formatted at 2026-09-24 04:16:06 on dist-test-slave-p401"
server_key: "0a166328244462f19bc34b0b0ca78a0d"
server_key_iv: "b8236a530c97ce26f7c235d95c87c899"
server_key_version: "kuduclusterkey@0"
I20260924 04:16:07.889572 13496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:16:07.908217 13496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:16:07.911810 13496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:16:07.912071 13496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:16:07.912807 13496 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:16:07.913842 13496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:16:07.913952 13496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:16:07.914057 13496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:16:07.914117 13496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:16:07.924820 13496 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43629
I20260924 04:16:07.924990 13626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43629 every 8 connection(s)
I20260924 04:16:07.925940 13496 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:16:07.930266 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 13496
Sep 24 04:16:07 dist-test-slave-p401 krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223367, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:16:07.942552 13362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:07.927786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53491 (local address 127.0.98.62:39963)
0924 04:16:07.927937 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:07.927941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:07.929457 (+ 1516us) server_negotiation.cc:407] Connection header received
0924 04:16:07.930631 (+ 1174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:07.930635 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:07.930691 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:07.930748 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:16:07.932371 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:07.932999 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:07.934656 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:07.934997 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:07.935077 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:07.938761 (+ 3684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:16:07.938765 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:16:07.938768 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:16:07.938793 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:16:07.940277 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:07.940854 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:07.940857 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:07.940859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:07.940945 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:07.941346 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:07.941349 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:07.941351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:07.941547 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:16:07.941616 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:07.942286 (+ 670us) server_negotiation.cc:299] Negotiation successful
0924 04:16:07.942398 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:16:07.944465 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:07.928168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39963 (local address 127.0.98.3:53491)
0924 04:16:07.929299 (+ 1131us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:07.929354 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 04:16:07.930388 (+ 1034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:16:07.931157 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:16:07.931166 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:16:07.931566 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:16:07.932201 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:07.932219 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:07.933152 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:07.933155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:07.934516 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:07.934528 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:07.935308 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:07.935312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:07.935501 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:07.936213 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:16:07.936233 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:16:07.938574 (+ 2341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:16:07.940384 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:07.940390 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:07.940405 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:07.940741 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:07.941051 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:07.941053 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:07.941055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:07.941238 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:07.941700 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:16:07.941706 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:16:07.942026 (+ 320us) client_negotiation.cc:769] Sending connection context
0924 04:16:07.942988 (+ 962us) client_negotiation.cc:240] Negotiation successful
0924 04:16:07.943344 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":886,"thread_start_us":118,"threads_started":1}
I20260924 04:16:07.946555 13627 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39963
I20260924 04:16:07.946790 13627 heartbeater.cc:464] Registering TS with master...
I20260924 04:16:07.947372 13627 heartbeater.cc:511] Master 127.0.98.62:39963 requested a full tablet report, sending...
W20260924 04:16:07.948968 13627 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39963 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 04:16:07.949079 13627 heartbeater.cc:464] Registering TS with master...
I20260924 04:16:07.949241 13627 heartbeater.cc:511] Master 127.0.98.62:39963 requested a full tablet report, sending...
I20260924 04:16:07.950037 13627 heartbeater.cc:464] Registering TS with master...
I20260924 04:16:07.950199 13627 heartbeater.cc:511] Master 127.0.98.62:39963 requested a full tablet report, sending...
W20260924 04:16:07.950929 13627 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 04:16:07.951766 13631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:07.931295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34944 (local address 127.0.98.62:39963)
0924 04:16:07.936983 (+ 5688us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:07.936987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:07.937000 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:16:07.937033 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:07.937036 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:07.937233 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:07.937322 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:16:07.939100 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:07.940791 (+ 1691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:07.941971 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:07.942296 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:07.942376 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:07.943918 (+ 1542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:16:07.943922 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:16:07.943924 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:16:07.943946 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:16:07.945806 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:07.950526 (+ 4720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:07.950529 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:07.950530 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:07.950613 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:07.950904 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:07.950907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:07.950909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:07.951117 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:16:07.951301 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:07.951536 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 04:16:07.951635 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5617,"thread_start_us":73,"threads_started":1}
Sep 24 04:16:08 dist-test-slave-p401 krb5kdc[12165](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790223368, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:16:08.464789 13359 heartbeater.cc:464] Registering TS with master...
I20260924 04:16:08.465090 13359 heartbeater.cc:511] Master 127.0.98.62:39963 requested a full tablet report, sending...
I20260924 04:16:08.466235 13161 ts_manager.cc:194] Registered new tserver with Master: ba934cbd2b56481cbc1743baffe26a49 (127.0.98.1:35115)
I20260924 04:16:08.467936 13161 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38367
Sep 24 04:16:08 dist-test-slave-p401 krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223368, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:16:08.708854 13493 heartbeater.cc:464] Registering TS with master...
I20260924 04:16:08.709158 13493 heartbeater.cc:511] Master 127.0.98.62:39963 requested a full tablet report, sending...
I20260924 04:16:08.709933 13161 ts_manager.cc:194] Registered new tserver with Master: d61a304adedb490891eb590885498ca0 (127.0.98.2:34937)
I20260924 04:16:08.711084 13161 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42749
Sep 24 04:16:08 dist-test-slave-p401 krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223368, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:16:08.951146 13627 heartbeater.cc:464] Registering TS with master...
I20260924 04:16:08.951509 13627 heartbeater.cc:511] Master 127.0.98.62:39963 requested a full tablet report, sending...
I20260924 04:16:08.952353 13161 ts_manager.cc:194] Registered new tserver with Master: 9555a575bb054c34b1ebfae7e91303a4 (127.0.98.3:43629)
I20260924 04:16:08.953156 13161 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53491
I20260924 04:16:08.957182 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:16:08.973421 13651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:08.962732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34954 (local address 127.0.98.62:39963)
0924 04:16:08.963100 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:08.963104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:08.963118 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:16:08.963164 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:08.963167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:08.963216 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:08.963333 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:16:08.965183 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:08.966002 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:08.967169 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:08.967511 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:08.967611 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:08.969307 (+ 1696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:16:08.969323 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:16:08.969326 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:16:08.969356 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:16:08.971082 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:08.971619 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:08.971622 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:08.971624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:08.971711 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:08.971984 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:08.971988 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:08.971989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:08.972200 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:16:08.972340 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:08.973104 (+ 764us) server_negotiation.cc:299] Negotiation successful
0924 04:16:08.973229 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":86,"threads_started":1}
I20260924 04:16:08.977488 13161 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34954:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:16:08.981489 13161 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:16:09.019253 13656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:08.995298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43629 (local address 127.0.0.1:47518)
0924 04:16:08.999408 (+ 4110us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:08.999423 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:16:08.999536 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:16:09.000372 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:16:09.000378 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:16:09.000402 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:16:09.000543 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:09.000550 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:09.002025 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:09.002028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:09.010501 (+ 8473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:09.010516 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:09.011801 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:09.011805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:09.011947 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:09.012119 (+ 172us) client_negotiation.cc:769] Sending connection context
0924 04:16:09.018834 (+ 6715us) client_negotiation.cc:240] Negotiation successful
0924 04:16:09.019050 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4039,"thread_start_us":101,"threads_started":1}
I20260924 04:16:09.019969 13659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:08.995892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47518 (local address 127.0.98.3:43629)
0924 04:16:08.997368 (+ 1476us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:08.997376 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:08.999639 (+ 2263us) server_negotiation.cc:407] Connection header received
0924 04:16:08.999731 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:08.999737 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:08.999999 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:09.000159 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:16:09.000700 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:09.001888 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:09.010686 (+ 8798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:09.011656 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:09.011773 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:09.011943 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:09.019666 (+ 7723us) server_negotiation.cc:299] Negotiation successful
0924 04:16:09.019824 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1370,"thread_start_us":87,"threads_started":1}
I20260924 04:16:09.022123 13657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:08.993857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41742 (local address 127.0.98.2:34937)
0924 04:16:09.000847 (+ 6990us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:09.000855 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:09.000878 (+ 23us) server_negotiation.cc:407] Connection header received
0924 04:16:09.000951 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:09.000956 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:09.001162 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:09.001306 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:16:09.007654 (+ 6348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:09.008942 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:09.010582 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:09.011653 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:09.011776 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:09.011949 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:09.021822 (+ 9873us) server_negotiation.cc:299] Negotiation successful
0924 04:16:09.021966 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6866,"thread_start_us":130,"threads_started":1}
I20260924 04:16:09.024953 13660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:09.003696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35115 (local address 127.0.0.1:40166)
0924 04:16:09.019664 (+ 15968us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:09.019680 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:16:09.019789 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:16:09.020652 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:16:09.020656 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:16:09.020679 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:16:09.020809 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:09.020816 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:09.022196 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:09.022199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:09.023303 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:09.023322 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:09.024540 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:09.024544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:09.024648 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:09.024671 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 04:16:09.024736 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 04:16:09.024809 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15896,"thread_start_us":128,"threads_started":1}
I20260924 04:16:09.025410 13661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:09.004145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40166 (local address 127.0.98.1:35115)
0924 04:16:09.019494 (+ 15349us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:09.019502 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:09.019897 (+ 395us) server_negotiation.cc:407] Connection header received
0924 04:16:09.019994 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:09.020009 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:09.020274 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:09.020457 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:16:09.020948 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:09.022075 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:09.023448 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:09.024218 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:09.024328 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:09.024476 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:09.025161 (+ 685us) server_negotiation.cc:299] Negotiation successful
0924 04:16:09.025265 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15207,"thread_start_us":113,"threads_started":1}
I20260924 04:16:09.025491 13542 tablet_service.cc:1467] Processing CreateTablet for tablet 375edc6759494addb13b07672ada0bcd (DEFAULT_TABLE table=test-table [id=5ba5e96618754121bc09af8a34817008]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:16:09.026734 13542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 375edc6759494addb13b07672ada0bcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:16:09.030472 13275 tablet_service.cc:1467] Processing CreateTablet for tablet 375edc6759494addb13b07672ada0bcd (DEFAULT_TABLE table=test-table [id=5ba5e96618754121bc09af8a34817008]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:16:09.031661 13275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 375edc6759494addb13b07672ada0bcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:16:09.021946 13658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:08.993718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34937 (local address 127.0.0.1:41742)
0924 04:16:08.996907 (+ 3189us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:08.996967 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 04:16:08.997096 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:16:09.007331 (+ 10235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:16:09.007335 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:16:09.007367 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:16:09.007475 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:09.007482 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:09.009108 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:09.009112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:09.010402 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:09.010417 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:09.021501 (+ 11084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:09.021504 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:09.021616 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:09.021645 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 04:16:09.021728 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 04:16:09.021799 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2978,"thread_start_us":148,"threads_started":1}
I20260924 04:16:09.038667 13424 tablet_service.cc:1467] Processing CreateTablet for tablet 375edc6759494addb13b07672ada0bcd (DEFAULT_TABLE table=test-table [id=5ba5e96618754121bc09af8a34817008]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:16:09.039902 13424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 375edc6759494addb13b07672ada0bcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:16:09.051729 13662 tablet_bootstrap.cc:492] T 375edc6759494addb13b07672ada0bcd P 9555a575bb054c34b1ebfae7e91303a4: Bootstrap starting.
I20260924 04:16:09.054185 13662 tablet_bootstrap.cc:654] T 375edc6759494addb13b07672ada0bcd P 9555a575bb054c34b1ebfae7e91303a4: Neither blocks nor log segments found. Creating new log.
I20260924 04:16:09.054522 13663 tablet_bootstrap.cc:492] T 375edc6759494addb13b07672ada0bcd P d61a304adedb490891eb590885498ca0: Bootstrap starting.
I20260924 04:16:09.055105 13662 log.cc:824] T 375edc6759494addb13b07672ada0bcd P 9555a575bb054c34b1ebfae7e91303a4: Log is configured to *not* fsync() on all Append() calls
I20260924 04:16:09.056138 13664 tablet_bootstrap.cc:492] T 375edc6759494addb13b07672ada0bcd P ba934cbd2b56481cbc1743baffe26a49: Bootstrap starting.
I20260924 04:16:09.057463 13663 tablet_bootstrap.cc:654] T 375edc6759494addb13b07672ada0bcd P d61a304adedb490891eb590885498ca0: Neither blocks nor log segments found. Creating new log.
I20260924 04:16:09.058185 13664 tablet_bootstrap.cc:654] T 375edc6759494addb13b07672ada0bcd P ba934cbd2b56481cbc1743baffe26a49: Neither blocks nor log segments found. Creating new log.
I20260924 04:16:09.058509 13663 log.cc:824] T 375edc6759494addb13b07672ada0bcd P d61a304adedb490891eb590885498ca0: Log is configured to *not* fsync() on all Append() calls
I20260924 04:16:09.058959 13664 log.cc:824] T 375edc6759494addb13b07672ada0bcd P ba934cbd2b56481cbc1743baffe26a49: Log is configured to *not* fsync() on all Append() calls
I20260924 04:16:09.061153 13662 tablet_bootstrap.cc:492] T 375edc6759494addb13b07672ada0bcd P 9555a575bb054c34b1ebfae7e91303a4: No bootstrap required, opened a new log
I20260924 04:16:09.061369 13662 ts_tablet_manager.cc:1397] T 375edc6759494addb13b07672ada0bcd P 9555a575bb054c34b1ebfae7e91303a4: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260924 04:16:09.061679 13663 tablet_bootstrap.cc:492] T 375edc6759494addb13b07672ada0bcd P d61a304adedb490891eb590885498ca0: No bootstrap required, opened a new log
I20260924 04:16:09.061866 13663 ts_tablet_manager.cc:1397] T 375edc6759494addb13b07672ada0bcd P d61a304adedb490891eb590885498ca0: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260924 04:16:09.065609 13662 raft_consensus.cc:348] T 375edc6759494addb13b07672ada0bcd P 9555a575bb054c34b1ebfae7e91303a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d61a304adedb490891eb590885498ca0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34937 } } peers { permanent_uuid: "9555a575bb054c34b1ebfae7e91303a4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43629 } } peers { permanent_uuid: "ba934cbd2b56481cbc1743baffe26a49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35115 } }
I20260924 04:16:09.065613 13663 raft_consensus.cc:348] T 375edc6759494addb13b07672ada0bcd P d61a304adedb490891eb590885498ca0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d61a304adedb490891eb590885498ca0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34937 } } peers { permanent_uuid: "9555a575bb054c34b1ebfae7e91303a4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43629 } } peers { permanent_uuid: "ba934cbd2b56481cbc1743baffe26a49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35115 } }
I20260924 04:16:09.065882 13663 raft_consensus.cc:374] T 375edc6759494addb13b07672ada0bcd P d61a304adedb490891eb590885498ca0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:16:09.065882 13662 raft_consensus.cc:374] T 375edc6759494addb13b07672ada0bcd P 9555a575bb054c34b1ebfae7e91303a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:16:09.065932 13662 raft_consensus.cc:729] T 375edc6759494addb13b07672ada0bcd P 9555a575bb054c34b1ebfae7e91303a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9555a575bb054c34b1ebfae7e91303a4, State: Initialized, Role: FOLLOWER
I20260924 04:16:09.065932 13663 raft_consensus.cc:729] T 375edc6759494addb13b07672ada0bcd P d61a304adedb490891eb590885498ca0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d61a304adedb490891eb590885498ca0, State: Initialized, Role: FOLLOWER
I20260924 04:16:09.066455 13662 consensus_queue.cc:260] T 375edc6759494addb13b07672ada0bcd P 9555a575bb054c34b1ebfae7e91303a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d61a304adedb490891eb590885498ca0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34937 } } peers { permanent_uuid: "9555a575bb054c34b1ebfae7e91303a4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43629 } } peers { permanent_uuid: "ba934cbd2b56481cbc1743baffe26a49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35115 } }
I20260924 04:16:09.066915 13663 consensus_queue.cc:260] T 375edc6759494addb13b07672ada0bcd P d61a304adedb490891eb590885498ca0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d61a304adedb490891eb590885498ca0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34937 } } peers { permanent_uuid: "9555a575bb054c34b1ebfae7e91303a4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43629 } } peers { permanent_uuid: "ba934cbd2b56481cbc1743baffe26a49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35115 } }
I20260924 04:16:09.067608 13662 ts_tablet_manager.cc:1428] T 375edc6759494addb13b07672ada0bcd P 9555a575bb054c34b1ebfae7e91303a4: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 04:16:09.067986 13627 heartbeater.cc:503] Master 127.0.98.62:39963 was elected leader, sending a full tablet report...
I20260924 04:16:09.069039 13493 heartbeater.cc:503] Master 127.0.98.62:39963 was elected leader, sending a full tablet report...
I20260924 04:16:09.070328 13664 tablet_bootstrap.cc:492] T 375edc6759494addb13b07672ada0bcd P ba934cbd2b56481cbc1743baffe26a49: No bootstrap required, opened a new log
I20260924 04:16:09.070533 13664 ts_tablet_manager.cc:1397] T 375edc6759494addb13b07672ada0bcd P ba934cbd2b56481cbc1743baffe26a49: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.003s
I20260924 04:16:09.074679 13664 raft_consensus.cc:348] T 375edc6759494addb13b07672ada0bcd P ba934cbd2b56481cbc1743baffe26a49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d61a304adedb490891eb590885498ca0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34937 } } peers { permanent_uuid: "9555a575bb054c34b1ebfae7e91303a4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43629 } } peers { permanent_uuid: "ba934cbd2b56481cbc1743baffe26a49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35115 } }
I20260924 04:16:09.074977 13664 raft_consensus.cc:374] T 375edc6759494addb13b07672ada0bcd P ba934cbd2b56481cbc1743baffe26a49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:16:09.075034 13664 raft_consensus.cc:729] T 375edc6759494addb13b07672ada0bcd P ba934cbd2b56481cbc1743baffe26a49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba934cbd2b56481cbc1743baffe26a49, State: Initialized, Role: FOLLOWER
I20260924 04:16:09.075528 13664 consensus_queue.cc:260] T 375edc6759494addb13b07672ada0bcd P ba934cbd2b56481cbc1743baffe26a49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d61a304adedb490891eb590885498ca0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34937 } } peers { permanent_uuid: "9555a575bb054c34b1ebfae7e91303a4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43629 } } peers { permanent_uuid: "ba934cbd2b56481cbc1743baffe26a49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35115 } }
I20260924 04:16:09.076467 13663 ts_tablet_manager.cc:1428] T 375edc6759494addb13b07672ada0bcd P d61a304adedb490891eb590885498ca0: Time spent starting tablet: real 0.015s user 0.005s sys 0.001s
I20260924 04:16:09.076597 13664 ts_tablet_manager.cc:1428] T 375edc6759494addb13b07672ada0bcd P ba934cbd2b56481cbc1743baffe26a49: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 04:16:09.076961 13359 heartbeater.cc:503] Master 127.0.98.62:39963 was elected leader, sending a full tablet report...
I20260924 04:16:09.166862 13668 raft_consensus.cc:482] T 375edc6759494addb13b07672ada0bcd P 9555a575bb054c34b1ebfae7e91303a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:16:09.167114 13668 raft_consensus.cc:504] T 375edc6759494addb13b07672ada0bcd P 9555a575bb054c34b1ebfae7e91303a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d61a304adedb490891eb590885498ca0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34937 } } peers { permanent_uuid: "9555a575bb054c34b1ebfae7e91303a4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43629 } } peers { permanent_uuid: "ba934cbd2b56481cbc1743baffe26a49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35115 } }
I20260924 04:16:09.168586 13668 leader_election.cc:288] T 375edc6759494addb13b07672ada0bcd P 9555a575bb054c34b1ebfae7e91303a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d61a304adedb490891eb590885498ca0 (127.0.98.2:34937), ba934cbd2b56481cbc1743baffe26a49 (127.0.98.1:35115)
I20260924 04:16:09.174237 13657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:09.169087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38973 (local address 127.0.98.2:34937)
0924 04:16:09.169216 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:09.169221 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:09.169364 (+ 143us) server_negotiation.cc:407] Connection header received
0924 04:16:09.169870 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:09.169874 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:09.169926 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:09.169971 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:16:09.170372 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:09.171177 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:09.172902 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:09.173518 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:09.173595 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:09.173661 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:09.174021 (+ 360us) server_negotiation.cc:299] Negotiation successful
0924 04:16:09.174105 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 04:16:09.175909 13671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:09.168993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34937 (local address 127.0.98.3:38973)
0924 04:16:09.169286 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:09.169300 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:16:09.169712 (+ 412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:16:09.170103 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:16:09.170106 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:16:09.170123 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:16:09.170257 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:09.170262 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:09.171360 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:09.171364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:09.172748 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:09.172759 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:09.173752 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:09.173755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:09.173874 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:09.173899 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 04:16:09.175668 (+ 1769us) client_negotiation.cc:240] Negotiation successful
0924 04:16:09.175753 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":106,"threads_started":1}
I20260924 04:16:09.176709 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "375edc6759494addb13b07672ada0bcd" candidate_uuid: "9555a575bb054c34b1ebfae7e91303a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d61a304adedb490891eb590885498ca0" is_pre_election: true
I20260924 04:16:09.177058 13447 raft_consensus.cc:2454] T 375edc6759494addb13b07672ada0bcd P d61a304adedb490891eb590885498ca0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9555a575bb054c34b1ebfae7e91303a4 in term 0.
I20260924 04:16:09.177637 13516 leader_election.cc:302] T 375edc6759494addb13b07672ada0bcd P 9555a575bb054c34b1ebfae7e91303a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9555a575bb054c34b1ebfae7e91303a4, d61a304adedb490891eb590885498ca0; no voters:
I20260924 04:16:09.177953 13668 raft_consensus.cc:2781] T 375edc6759494addb13b07672ada0bcd P 9555a575bb054c34b1ebfae7e91303a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:16:09.178020 13668 raft_consensus.cc:482] T 375edc6759494addb13b07672ada0bcd P 9555a575bb054c34b1ebfae7e91303a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:16:09.178056 13668 raft_consensus.cc:3037] T 375edc6759494addb13b07672ada0bcd P 9555a575bb054c34b1ebfae7e91303a4 [term 0 FOLLOWER]: Advancing to term 1
W20260924 04:16:09.178331 13628 tablet.cc:2369] T 375edc6759494addb13b07672ada0bcd P 9555a575bb054c34b1ebfae7e91303a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:16:09.179498 13668 raft_consensus.cc:504] T 375edc6759494addb13b07672ada0bcd P 9555a575bb054c34b1ebfae7e91303a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d61a304adedb490891eb590885498ca0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34937 } } peers { permanent_uuid: "9555a575bb054c34b1ebfae7e91303a4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43629 } } peers { permanent_uuid: "ba934cbd2b56481cbc1743baffe26a49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35115 } }
I20260924 04:16:09.179841 13668 leader_election.cc:288] T 375edc6759494addb13b07672ada0bcd P 9555a575bb054c34b1ebfae7e91303a4 [CANDIDATE]: Term 1 election: Requested vote from peers d61a304adedb490891eb590885498ca0 (127.0.98.2:34937), ba934cbd2b56481cbc1743baffe26a49 (127.0.98.1:35115)
I20260924 04:16:09.180315 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "375edc6759494addb13b07672ada0bcd" candidate_uuid: "9555a575bb054c34b1ebfae7e91303a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d61a304adedb490891eb590885498ca0"
I20260924 04:16:09.180426 13447 raft_consensus.cc:3037] T 375edc6759494addb13b07672ada0bcd P d61a304adedb490891eb590885498ca0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:16:09.181608 13447 raft_consensus.cc:2454] T 375edc6759494addb13b07672ada0bcd P d61a304adedb490891eb590885498ca0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9555a575bb054c34b1ebfae7e91303a4 in term 1.
I20260924 04:16:09.181954 13516 leader_election.cc:302] T 375edc6759494addb13b07672ada0bcd P 9555a575bb054c34b1ebfae7e91303a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9555a575bb054c34b1ebfae7e91303a4, d61a304adedb490891eb590885498ca0; no voters:
I20260924 04:16:09.182144 13668 raft_consensus.cc:2781] T 375edc6759494addb13b07672ada0bcd P 9555a575bb054c34b1ebfae7e91303a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:16:09.182329 13668 raft_consensus.cc:686] T 375edc6759494addb13b07672ada0bcd P 9555a575bb054c34b1ebfae7e91303a4 [term 1 LEADER]: Becoming Leader. State: Replica: 9555a575bb054c34b1ebfae7e91303a4, State: Running, Role: LEADER
I20260924 04:16:09.182657 13668 consensus_queue.cc:237] T 375edc6759494addb13b07672ada0bcd P 9555a575bb054c34b1ebfae7e91303a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d61a304adedb490891eb590885498ca0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34937 } } peers { permanent_uuid: "9555a575bb054c34b1ebfae7e91303a4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43629 } } peers { permanent_uuid: "ba934cbd2b56481cbc1743baffe26a49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35115 } }
W20260924 04:16:09.183437 13360 tablet.cc:2369] T 375edc6759494addb13b07672ada0bcd P ba934cbd2b56481cbc1743baffe26a49: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:16:09.186012 13161 catalog_manager.cc:5495] T 375edc6759494addb13b07672ada0bcd P 9555a575bb054c34b1ebfae7e91303a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9555a575bb054c34b1ebfae7e91303a4 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "9555a575bb054c34b1ebfae7e91303a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d61a304adedb490891eb590885498ca0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9555a575bb054c34b1ebfae7e91303a4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ba934cbd2b56481cbc1743baffe26a49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35115 } health_report { overall_health: UNKNOWN } } }
W20260924 04:16:09.188050 13494 tablet.cc:2369] T 375edc6759494addb13b07672ada0bcd P d61a304adedb490891eb590885498ca0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:16:09.193485 13661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:09.169521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35541 (local address 127.0.98.1:35115)
0924 04:16:09.169636 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:09.169640 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:09.189112 (+ 19472us) server_negotiation.cc:407] Connection header received
0924 04:16:09.189316 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:09.189320 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:09.189380 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:09.189431 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:16:09.189827 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:09.190597 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:09.192096 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:09.192716 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:09.192800 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:09.192883 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:09.193253 (+ 370us) server_negotiation.cc:299] Negotiation successful
0924 04:16:09.193333 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:16:09.194126 13671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:09.169388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35115 (local address 127.0.98.3:35541)
0924 04:16:09.189029 (+ 19641us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:09.189045 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:16:09.189185 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:16:09.189573 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:16:09.189576 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:16:09.189594 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:16:09.189682 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:09.189688 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:09.190738 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:09.190741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:09.191952 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:09.191963 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:09.193015 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:09.193018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:09.193109 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:09.193132 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 04:16:09.193870 (+ 738us) client_negotiation.cc:240] Negotiation successful
0924 04:16:09.193949 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18790,"thread_start_us":97,"threads_started":1}
I20260924 04:16:09.195379 13295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "375edc6759494addb13b07672ada0bcd" candidate_uuid: "9555a575bb054c34b1ebfae7e91303a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba934cbd2b56481cbc1743baffe26a49"
I20260924 04:16:09.195609 13295 raft_consensus.cc:3037] T 375edc6759494addb13b07672ada0bcd P ba934cbd2b56481cbc1743baffe26a49 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:16:09.197124 13295 raft_consensus.cc:2454] T 375edc6759494addb13b07672ada0bcd P ba934cbd2b56481cbc1743baffe26a49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9555a575bb054c34b1ebfae7e91303a4 in term 1.
I20260924 04:16:09.195173 13313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "375edc6759494addb13b07672ada0bcd" candidate_uuid: "9555a575bb054c34b1ebfae7e91303a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba934cbd2b56481cbc1743baffe26a49" is_pre_election: true
I20260924 04:16:09.197751 13313 raft_consensus.cc:2362] T 375edc6759494addb13b07672ada0bcd P ba934cbd2b56481cbc1743baffe26a49 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 9555a575bb054c34b1ebfae7e91303a4 in term 1. Re-sending same reply.
I20260924 04:16:09.222867 13659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:09.217921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47524 (local address 127.0.98.3:43629)
0924 04:16:09.218036 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:09.218040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:09.218052 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:16:09.218135 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:09.218138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:09.218208 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:09.218319 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:16:09.218736 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:09.219564 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:09.220768 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:09.221115 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:09.221208 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:09.221614 (+ 406us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:16:09.222170 (+ 556us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:16:09.222281 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:09.222656 (+ 375us) server_negotiation.cc:299] Negotiation successful
0924 04:16:09.222714 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 04:16:09.229532 13313 raft_consensus.cc:1261] T 375edc6759494addb13b07672ada0bcd P ba934cbd2b56481cbc1743baffe26a49 [term 1 FOLLOWER]: Refusing update from remote peer 9555a575bb054c34b1ebfae7e91303a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:16:09.230432 13673 consensus_queue.cc:1035] T 375edc6759494addb13b07672ada0bcd P 9555a575bb054c34b1ebfae7e91303a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba934cbd2b56481cbc1743baffe26a49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:16:09.237561 13447 raft_consensus.cc:1261] T 375edc6759494addb13b07672ada0bcd P d61a304adedb490891eb590885498ca0 [term 1 FOLLOWER]: Refusing update from remote peer 9555a575bb054c34b1ebfae7e91303a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:16:09.238389 13673 consensus_queue.cc:1035] T 375edc6759494addb13b07672ada0bcd P 9555a575bb054c34b1ebfae7e91303a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d61a304adedb490891eb590885498ca0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:16:09.253718 13676 mvcc.cc:205] Tried to move back new op lower bound from 7332754920355319808 to 7332754920175480832. Current Snapshot: MvccSnapshot[applied={T|T < 7332754920355319808}]
I20260924 04:16:09.263441 13679 mvcc.cc:205] Tried to move back new op lower bound from 7332754920355319808 to 7332754920175480832. Current Snapshot: MvccSnapshot[applied={T|T < 7332754920355319808}]
I20260924 04:16:09.265743 13677 mvcc.cc:205] Tried to move back new op lower bound from 7332754920355319808 to 7332754920175480832. Current Snapshot: MvccSnapshot[applied={T|T < 7332754920355319808}]
I20260924 04:16:09.276131 13161 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:16:09.276331 13161 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:16:09.279332 13161 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3bb47c28dd6b41769a35cef4cf70f3f7: Sending DeleteTablet for 3 replicas of tablet 375edc6759494addb13b07672ada0bcd
I20260924 04:16:09.280472 13424 tablet_service.cc:1514] Processing DeleteTablet for tablet 375edc6759494addb13b07672ada0bcd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:16:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41742
I20260924 04:16:09.281195 13542 tablet_service.cc:1514] Processing DeleteTablet for tablet 375edc6759494addb13b07672ada0bcd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:16:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47518
I20260924 04:16:09.281590 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 13199
I20260924 04:16:09.287447 13692 tablet_replica.cc:330] T 375edc6759494addb13b07672ada0bcd P 9555a575bb054c34b1ebfae7e91303a4: stopping tablet replica
I20260924 04:16:09.287530 13691 tablet_replica.cc:330] T 375edc6759494addb13b07672ada0bcd P d61a304adedb490891eb590885498ca0: stopping tablet replica
I20260924 04:16:09.287874 13692 raft_consensus.cc:2229] T 375edc6759494addb13b07672ada0bcd P 9555a575bb054c34b1ebfae7e91303a4 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:16:09.287948 13691 raft_consensus.cc:2229] T 375edc6759494addb13b07672ada0bcd P d61a304adedb490891eb590885498ca0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:16:09.288337 13692 raft_consensus.cc:2258] T 375edc6759494addb13b07672ada0bcd P 9555a575bb054c34b1ebfae7e91303a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:16:09.288352 13691 raft_consensus.cc:2258] T 375edc6759494addb13b07672ada0bcd P d61a304adedb490891eb590885498ca0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:16:09.289827 13691 ts_tablet_manager.cc:1905] T 375edc6759494addb13b07672ada0bcd P d61a304adedb490891eb590885498ca0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:16:09.289827 13692 ts_tablet_manager.cc:1905] T 375edc6759494addb13b07672ada0bcd P 9555a575bb054c34b1ebfae7e91303a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:16:09.292624 13692 ts_tablet_manager.cc:1918] T 375edc6759494addb13b07672ada0bcd P 9555a575bb054c34b1ebfae7e91303a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:16:09.293411 13691 ts_tablet_manager.cc:1918] T 375edc6759494addb13b07672ada0bcd P d61a304adedb490891eb590885498ca0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:16:09.293509 13691 log.cc:1196] T 375edc6759494addb13b07672ada0bcd P d61a304adedb490891eb590885498ca0: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790223216265354-392-0/minicluster-data/ts-1/wal/wals/375edc6759494addb13b07672ada0bcd
W20260924 04:16:09.293686 13147 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35115 (error 108)
I20260924 04:16:09.293797 13691 ts_tablet_manager.cc:1939] T 375edc6759494addb13b07672ada0bcd P d61a304adedb490891eb590885498ca0: Deleting consensus metadata
I20260924 04:16:09.295173 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 13363
I20260924 04:16:09.295190 13656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:09.294925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35115 (local address 127.0.0.1:40174)
0924 04:16:09.295034 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:09.295102 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 04:16:09.295876 13147 catalog_manager.cc:4584] TS ba934cbd2b56481cbc1743baffe26a49 (127.0.98.1:35115): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 375edc6759494addb13b07672ada0bcd: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35115: connect: Connection refused (error 111)
I20260924 04:16:09.301487 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 13496
I20260924 04:16:09.301553 13656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:09.301267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34937 (local address 127.0.0.1:41758)
0924 04:16:09.301411 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:09.301466 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 04:16:09.310230 13656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:09.309956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43629 (local address 127.0.0.1:47538)
0924 04:16:09.310082 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:09.310134 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.3:43629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 04:16:09.310338 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 13128
I20260924 04:16:09.315996 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 04:16:09.650839 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 04:16:09.650947 392 mini_ranger.cc:66] Stopping Ranger...
I20260924 04:16:09.985949 392 mini_ranger.cc:68] Stopped Ranger
2026-09-24 04:16:09.986 UTC [12195] LOG: received smart shutdown request
2026-09-24 04:16:09.989 UTC [12195] LOG: background worker "logical replication launcher" (PID 12204) exited with exit code 1
2026-09-24 04:16:09.989 UTC [12199] LOG: shutting down
2026-09-24 04:16:10.164 UTC [12195] LOG: database system is shut down
2026-09-24T04:16:10Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47746 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:16:10 dist-test-slave-p401 krb5kdc[13703](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:16:10 dist-test-slave-p401 krb5kdc[13703](info): set up 2 sockets
Sep 24 04:16:10 dist-test-slave-p401 krb5kdc[13703](info): commencing operation
krb5kdc: starting...
W20260924 04:16:11.227102 392 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/krb5kdc/test-user.keytab.
Sep 24 04:16:11 dist-test-slave-p401 krb5kdc[13703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:16:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:16:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:16:11.340298 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40549
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40549,127.0.98.61:36767,127.0.98.60:42381
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:16:11.445298 13716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:16:11.445550 13716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:16:11.445588 13716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:16:11.449503 13716 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:16:11.449554 13716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:16:11.449572 13716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:16:11.449587 13716 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:16:11.449601 13716 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:16:11.454712 13716 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40549,127.0.98.61:36767,127.0.98.60:42381
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:16:11.455724 13716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:16:11.456686 13716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:16:11.464887 13724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:16:11.465019 13716 server_base.cc:1034] running on GCE node
W20260924 04:16:11.464891 13722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:16:11.464910 13721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:16:11.465737 13716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:16:11.466832 13716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:16:11.468007 13716 hybrid_clock.cc:648] HybridClock initialized: now 1790223371467982 us; error 40 us; skew 500 ppm
Sep 24 04:16:11 dist-test-slave-p401 krb5kdc[13703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:16:11.470681 13716 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:16:11.471637 13716 webserver.cc:460] Webserver started at http://127.0.98.62:43145/ using document root <none> and password file <none>
I20260924 04:16:11.472160 13716 fs_manager.cc:359] Metadata directory not provided
I20260924 04:16:11.472224 13716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:16:11.472407 13716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:16:11.474365 13716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/minicluster-data/master-0/data/instance:
uuid: "a0939eed012f423a9ad29b650876a140"
format_stamp: "Formatted at 2026-09-24 04:16:11 on dist-test-slave-p401"
server_key: "0c1808ff40ad5866c13ddc1a1462f48e"
server_key_iv: "b05e8a778ea453ed6574ba307712d859"
server_key_version: "encryptionkey@0"
I20260924 04:16:11.474915 13716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/minicluster-data/master-0/wal/instance:
uuid: "a0939eed012f423a9ad29b650876a140"
format_stamp: "Formatted at 2026-09-24 04:16:11 on dist-test-slave-p401"
server_key: "0c1808ff40ad5866c13ddc1a1462f48e"
server_key_iv: "b05e8a778ea453ed6574ba307712d859"
server_key_version: "encryptionkey@0"
I20260924 04:16:11.478497 13716 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 04:16:11.480863 13731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:16:11.482076 13716 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:16:11.482198 13716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/minicluster-data/master-0/wal
uuid: "a0939eed012f423a9ad29b650876a140"
format_stamp: "Formatted at 2026-09-24 04:16:11 on dist-test-slave-p401"
server_key: "0c1808ff40ad5866c13ddc1a1462f48e"
server_key_iv: "b05e8a778ea453ed6574ba307712d859"
server_key_version: "encryptionkey@0"
I20260924 04:16:11.482298 13716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:16:11.518484 13716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:16:11.521279 13716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:16:11.521430 13716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:16:11.527871 13716 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40549
I20260924 04:16:11.527895 13783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40549 every 8 connection(s)
I20260924 04:16:11.528766 13716 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:16:11.531399 13784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:16:11.535036 13784 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40549 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:16:11.535884 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 13716
I20260924 04:16:11.536017 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/minicluster-data/master-0/wal/instance
I20260924 04:16:11.536285 392 external_mini_cluster.cc:1428] Setting key 263222d56a87724ceb17f6303e48dea4
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 24 04:16:11 dist-test-slave-p401 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:16:11.552065 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:11.537390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40549 (local address 127.0.0.1:52204)
0924 04:16:11.538061 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:11.538111 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 04:16:11.539102 (+ 991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:16:11.539838 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:16:11.539844 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:16:11.540306 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:16:11.540665 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:11.540683 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:11.541829 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:11.541832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:11.542970 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:11.542982 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:11.543693 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:11.543697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:11.543819 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:11.544395 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:16:11.544411 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:16:11.547533 (+ 3122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:16:11.549303 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:11.549313 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:11.549329 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:11.549597 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:11.549878 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:11.549880 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:11.549882 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:11.550016 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:11.550599 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:16:11.550606 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:16:11.550759 (+ 153us) client_negotiation.cc:769] Sending connection context
0924 04:16:11.550948 (+ 189us) client_negotiation.cc:240] Negotiation successful
0924 04:16:11.551312 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"mutex_wait_us":57,"spinlock_wait_cycles":34176,"thread_start_us":158,"threads_started":1}
I20260924 04:16:11.552066 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:11.537390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52204 (local address 127.0.98.62:40549)
0924 04:16:11.537804 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:11.537821 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:11.538220 (+ 399us) server_negotiation.cc:407] Connection header received
0924 04:16:11.539439 (+ 1219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:11.539444 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:11.539627 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:11.539803 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:16:11.540828 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:11.541700 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:11.543097 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:11.543535 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:11.543668 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:11.547672 (+ 4004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:16:11.547679 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:16:11.547692 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:16:11.547717 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:16:11.549194 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:11.549688 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:11.549693 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:11.549694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:11.549797 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:11.550092 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:11.550094 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:11.550095 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:11.550491 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:16:11.550590 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:11.550997 (+ 407us) server_negotiation.cc:299] Negotiation successful
0924 04:16:11.551250 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":283,"thread_start_us":121,"threads_started":1}
I20260924 04:16:11.554560 13784 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36767 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:16:11.555174 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:11.554957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36767 (local address 127.0.0.1:58126)
0924 04:16:11.555049 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:11.555118 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 04:16:11.555354 13733 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36767: connect: Connection refused (error 111)
I20260924 04:16:11.556247 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:11.556056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36767 (local address 127.0.0.1:58136)
0924 04:16:11.556159 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:11.556198 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 04:16:11.556397 13784 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36767: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36767: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260924 04:16:11.562675 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36767
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40549,127.0.98.61:36767,127.0.98.60:42381
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:16:11.605665 13784 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36767 } attempt: 1
I20260924 04:16:11.606563 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:11.606246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36767 (local address 127.0.0.1:58144)
0924 04:16:11.606402 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:11.606474 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 04:16:11.607323 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:11.607078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36767 (local address 127.0.0.1:58158)
0924 04:16:11.607170 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:11.607222 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 04:16:11.607483 13784 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36767: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36767: connect: Connection refused (error 111)
W20260924 04:16:11.667917 13791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:16:11.668174 13791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:16:11.668228 13791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:16:11.672161 13791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:16:11.672216 13791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:16:11.672243 13791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:16:11.672264 13791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:16:11.672294 13791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:16:11.675624 13784 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36767 } attempt: 2
I20260924 04:16:11.676425 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:11.676115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36767 (local address 127.0.0.1:58164)
0924 04:16:11.676283 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:11.676348 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260924 04:16:11.677066 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:11.676879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36767 (local address 127.0.0.1:58168)
0924 04:16:11.676965 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:11.677009 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 04:16:11.677218 13784 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36767: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36767: connect: Connection refused (error 111)
I20260924 04:16:11.677836 13791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40549,127.0.98.61:36767,127.0.98.60:42381
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:16:11.678826 13791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:16:11.679816 13791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:16:11.687759 13796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:16:11.687759 13791 server_base.cc:1034] running on GCE node
W20260924 04:16:11.687791 13797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:16:11.687788 13799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:16:11.688519 13791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:16:11.689670 13791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:16:11.690861 13791 hybrid_clock.cc:648] HybridClock initialized: now 1790223371690832 us; error 44 us; skew 500 ppm
Sep 24 04:16:11 dist-test-slave-p401 krb5kdc[13703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:16:11.693825 13791 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260924 04:16:11.694752 13791 webserver.cc:460] Webserver started at http://127.0.98.61:39129/ using document root <none> and password file <none>
I20260924 04:16:11.695364 13791 fs_manager.cc:359] Metadata directory not provided
I20260924 04:16:11.695441 13791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:16:11.695649 13791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:16:11.697716 13791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/minicluster-data/master-1/data/instance:
uuid: "739b14307d3e43199f304afbf39a29b3"
format_stamp: "Formatted at 2026-09-24 04:16:11 on dist-test-slave-p401"
server_key: "1b0aabbef884602ffa594862f94cb71a"
server_key_iv: "8e312a7f7da11963c4e59e4e30f55604"
server_key_version: "encryptionkey@0"
I20260924 04:16:11.698251 13791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/minicluster-data/master-1/wal/instance:
uuid: "739b14307d3e43199f304afbf39a29b3"
format_stamp: "Formatted at 2026-09-24 04:16:11 on dist-test-slave-p401"
server_key: "1b0aabbef884602ffa594862f94cb71a"
server_key_iv: "8e312a7f7da11963c4e59e4e30f55604"
server_key_version: "encryptionkey@0"
I20260924 04:16:11.701896 13791 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 04:16:11.704299 13806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:16:11.705487 13791 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:16:11.705597 13791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/minicluster-data/master-1/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/minicluster-data/master-1/wal
uuid: "739b14307d3e43199f304afbf39a29b3"
format_stamp: "Formatted at 2026-09-24 04:16:11 on dist-test-slave-p401"
server_key: "1b0aabbef884602ffa594862f94cb71a"
server_key_iv: "8e312a7f7da11963c4e59e4e30f55604"
server_key_version: "encryptionkey@0"
I20260924 04:16:11.705688 13791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:16:11.732120 13791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:16:11.734823 13791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:16:11.734995 13791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:16:11.741591 13791 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:36767
I20260924 04:16:11.741655 13858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:36767 every 8 connection(s)
I20260924 04:16:11.742523 13791 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/minicluster-data/master-1/data/info.pb
I20260924 04:16:11.745208 13859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:16:11.748317 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 13791
I20260924 04:16:11.748436 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/minicluster-data/master-1/wal/instance
I20260924 04:16:11.748677 392 external_mini_cluster.cc:1428] Setting key 31208194d2ae4a05d0736248d3669d30
I20260924 04:16:11.749039 13859 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40549 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 24 04:16:11 dist-test-slave-p401 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:16:11.762518 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:11.750811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52216 (local address 127.0.98.62:40549)
0924 04:16:11.750953 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:11.750958 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:11.751795 (+ 837us) server_negotiation.cc:407] Connection header received
0924 04:16:11.752752 (+ 957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:11.752756 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:11.752819 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:11.752939 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:16:11.753999 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:11.754812 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:11.755956 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:11.756186 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:11.756281 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:11.758884 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:16:11.758888 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:16:11.758890 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:16:11.758914 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:16:11.760504 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:11.761051 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:11.761054 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:11.761055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:11.761123 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:11.761477 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:11.761479 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:11.761480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:11.761639 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:16:11.761717 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:11.762225 (+ 508us) server_negotiation.cc:299] Negotiation successful
0924 04:16:11.762357 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:16:11.763412 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:11.751169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40549 (local address 127.0.0.1:52216)
0924 04:16:11.751652 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:11.751703 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 04:16:11.752539 (+ 836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:16:11.753064 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:16:11.753070 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:16:11.753397 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:16:11.753856 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:11.753869 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:11.754931 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:11.754933 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:11.755850 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:11.755860 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:11.756305 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:11.756307 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:11.756459 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:11.757007 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:16:11.757023 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:16:11.758704 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:16:11.760616 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:11.760622 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:11.760638 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:11.760965 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:11.761192 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:11.761195 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:11.761197 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:11.761399 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:11.761742 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:16:11.761748 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:16:11.761991 (+ 243us) client_negotiation.cc:769] Sending connection context
0924 04:16:11.762156 (+ 165us) client_negotiation.cc:240] Negotiation successful
0924 04:16:11.762474 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":107,"threads_started":1}
I20260924 04:16:11.764698 13859 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36767 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:16:11.768364 13784 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36767 } attempt: 3
Sep 24 04:16:11 dist-test-slave-p401 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260924 04:16:11.773627 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:42381
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40549,127.0.98.61:36767,127.0.98.60:42381
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 04:16:11 dist-test-slave-p401 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 04:16:11.774533 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:11.765148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36767 (local address 127.0.0.1:58174)
0924 04:16:11.765305 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:11.765315 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 04:16:11.765376 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:16:11.765912 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:16:11.765915 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:16:11.766075 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:16:11.766158 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:11.766163 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:11.767081 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:11.767084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:11.767648 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:11.767656 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:11.768199 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:11.768202 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:11.768343 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:11.768886 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:16:11.768902 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:16:11.770315 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:16:11.772714 (+ 2399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:11.772716 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:11.772718 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:11.773139 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:11.773466 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:11.773470 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:11.773472 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:11.773559 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:11.774093 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:16:11.774095 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:16:11.774167 (+ 72us) client_negotiation.cc:769] Sending connection context
0924 04:16:11.774249 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 04:16:11.774376 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":31}
I20260924 04:16:11.775388 13864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:11.765282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58174 (local address 127.0.98.61:36767)
0924 04:16:11.765587 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:11.765596 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:11.765613 (+ 17us) server_negotiation.cc:407] Connection header received
0924 04:16:11.765645 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:11.765649 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:11.765799 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:11.765916 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:16:11.766252 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:11.766985 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:11.767746 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:11.768099 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:11.768187 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:11.770456 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:16:11.770464 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:16:11.770476 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:16:11.770496 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:16:11.772491 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:11.773252 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:11.773258 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:11.773260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:11.773349 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:11.773655 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:11.773658 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:11.773660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:11.773944 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:16:11.774061 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:11.775040 (+ 979us) server_negotiation.cc:299] Negotiation successful
0924 04:16:11.775191 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":225,"thread_start_us":128,"threads_started":1}
I20260924 04:16:11.777354 13859 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42381 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:16:11.777760 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:11.768809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36767 (local address 127.0.0.1:58180)
0924 04:16:11.768935 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:11.768949 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:16:11.769015 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:16:11.769442 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:16:11.769445 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:16:11.769591 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:16:11.770016 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:11.770023 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:11.770847 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:11.770851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:11.771569 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:11.771581 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:11.772591 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:11.772595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:11.772698 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:11.773081 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:16:11.773094 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:16:11.774340 (+ 1246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:16:11.776144 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:11.776147 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:11.776149 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:11.776388 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:11.776844 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:11.776848 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:11.776850 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:11.776930 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:11.777288 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:16:11.777292 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:16:11.777389 (+ 97us) client_negotiation.cc:769] Sending connection context
0924 04:16:11.777483 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 04:16:11.777610 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 04:16:11.778148 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:11.777839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42381 (local address 127.0.0.1:50076)
0924 04:16:11.778004 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:11.778079 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260924 04:16:11.778324 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:11.768977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58180 (local address 127.0.98.61:36767)
0924 04:16:11.769259 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:11.769265 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:11.769275 (+ 10us) server_negotiation.cc:407] Connection header received
0924 04:16:11.769300 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:11.769302 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:11.769333 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:11.769402 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:16:11.770117 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:11.770750 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:11.771694 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:11.772460 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:11.772543 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:11.774468 (+ 1925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:16:11.774471 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:16:11.774473 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:16:11.774493 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:16:11.776023 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:11.776676 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:11.776680 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:11.776682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:11.776758 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:11.777025 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:11.777028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:11.777030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:11.777199 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:16:11.777280 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:11.778055 (+ 775us) server_negotiation.cc:299] Negotiation successful
0924 04:16:11.778143 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":94,"threads_started":1}
W20260924 04:16:11.778345 13808 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42381: connect: Connection refused (error 111)
I20260924 04:16:11.779448 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:11.779074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42381 (local address 127.0.0.1:50078)
0924 04:16:11.779290 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:11.779349 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":85,"threads_started":1}
I20260924 04:16:11.779464 13784 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42381 } has no permanent_uuid. Determining permanent_uuid...
W20260924 04:16:11.779633 13859 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42381: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42381: connect: Connection refused (error 111)
I20260924 04:16:11.780105 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:11.779893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42381 (local address 127.0.0.1:50086)
0924 04:16:11.780002 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:11.780046 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 04:16:11.780738 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:11.780515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42381 (local address 127.0.0.1:50090)
0924 04:16:11.780650 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:11.780688 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260924 04:16:11.780853 13784 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42381: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42381: connect: Connection refused (error 111)
I20260924 04:16:11.811987 13784 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42381 } attempt: 1
I20260924 04:16:11.812754 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:11.812478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42381 (local address 127.0.0.1:50106)
0924 04:16:11.812617 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:11.812677 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 04:16:11.813473 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:11.813284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42381 (local address 127.0.0.1:50116)
0924 04:16:11.813377 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:11.813420 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260924 04:16:11.813581 13784 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42381: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42381: connect: Connection refused (error 111)
I20260924 04:16:11.828871 13859 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42381 } attempt: 1
I20260924 04:16:11.829720 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:11.829398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42381 (local address 127.0.0.1:50128)
0924 04:16:11.829578 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:11.829640 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260924 04:16:11.830416 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:11.830187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42381 (local address 127.0.0.1:50138)
0924 04:16:11.830304 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:11.830360 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 04:16:11.830530 13859 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42381: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42381: connect: Connection refused (error 111)
W20260924 04:16:11.882936 13866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:16:11.883184 13866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:16:11.883222 13866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:16:11.887454 13866 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:16:11.887517 13866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:16:11.887535 13866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:16:11.887550 13866 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:16:11.887565 13866 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:16:11.888718 13784 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42381 } attempt: 2
I20260924 04:16:11.889494 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:11.889207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42381 (local address 127.0.0.1:50152)
0924 04:16:11.889358 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:11.889416 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 04:16:11.890136 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:11.889939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42381 (local address 127.0.0.1:50168)
0924 04:16:11.890036 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:11.890080 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 04:16:11.890237 13784 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42381: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42381: connect: Connection refused (error 111)
I20260924 04:16:11.893070 13866 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40549,127.0.98.61:36767,127.0.98.60:42381
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:42381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:16:11.894114 13866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:16:11.895094 13866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:16:11.898674 13859 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42381 } attempt: 2
I20260924 04:16:11.899531 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:11.899150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42381 (local address 127.0.0.1:50178)
0924 04:16:11.899385 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:11.899447 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
I20260924 04:16:11.900220 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:11.899996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42381 (local address 127.0.0.1:50186)
0924 04:16:11.900104 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:11.900156 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 04:16:11.900367 13859 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42381: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42381: connect: Connection refused (error 111)
W20260924 04:16:11.903502 13873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:16:11.903498 13874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:16:11.903687 13866 server_base.cc:1034] running on GCE node
W20260924 04:16:11.903750 13876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:16:11.904301 13866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:16:11.905378 13866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:16:11.906531 13866 hybrid_clock.cc:648] HybridClock initialized: now 1790223371906513 us; error 31 us; skew 500 ppm
Sep 24 04:16:11 dist-test-slave-p401 krb5kdc[13703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:16:11.909108 13866 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260924 04:16:11.910008 13866 webserver.cc:460] Webserver started at http://127.0.98.60:41133/ using document root <none> and password file <none>
I20260924 04:16:11.910530 13866 fs_manager.cc:359] Metadata directory not provided
I20260924 04:16:11.910593 13866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:16:11.910787 13866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:16:11.912792 13866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/minicluster-data/master-2/data/instance:
uuid: "6884677628724fb69810ce1816d18ab6"
format_stamp: "Formatted at 2026-09-24 04:16:11 on dist-test-slave-p401"
server_key: "4d72d67071f4ff64084428e5973cc001"
server_key_iv: "a0701fc1aaf0dab2b0665d657dac5219"
server_key_version: "encryptionkey@0"
I20260924 04:16:11.913311 13866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/minicluster-data/master-2/wal/instance:
uuid: "6884677628724fb69810ce1816d18ab6"
format_stamp: "Formatted at 2026-09-24 04:16:11 on dist-test-slave-p401"
server_key: "4d72d67071f4ff64084428e5973cc001"
server_key_iv: "a0701fc1aaf0dab2b0665d657dac5219"
server_key_version: "encryptionkey@0"
I20260924 04:16:11.916828 13866 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:16:11.919088 13883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:16:11.920311 13866 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:16:11.920425 13866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/minicluster-data/master-2/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/minicluster-data/master-2/wal
uuid: "6884677628724fb69810ce1816d18ab6"
format_stamp: "Formatted at 2026-09-24 04:16:11 on dist-test-slave-p401"
server_key: "4d72d67071f4ff64084428e5973cc001"
server_key_iv: "a0701fc1aaf0dab2b0665d657dac5219"
server_key_version: "encryptionkey@0"
I20260924 04:16:11.920516 13866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:16:11.956542 13866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:16:11.959753 13866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:16:11.959929 13866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:16:11.966936 13866 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:42381
I20260924 04:16:11.966987 13935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:42381 every 8 connection(s)
I20260924 04:16:11.967911 13866 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/minicluster-data/master-2/data/info.pb
I20260924 04:16:11.970472 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 13866
I20260924 04:16:11.970592 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790223216265354-392-0/minicluster-data/master-2/wal/instance
I20260924 04:16:11.970634 13936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:16:11.970860 392 external_mini_cluster.cc:1428] Setting key 6758fc5a5bded54e226e02cfbd16ea2b
I20260924 04:16:11.975040 13936 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40549 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 04:16:11 dist-test-slave-p401 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 04:16:11 dist-test-slave-p401 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:16:11.984730 13939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:11.972219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50192 (local address 127.0.98.60:42381)
0924 04:16:11.972615 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:11.972629 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:11.972663 (+ 34us) server_negotiation.cc:407] Connection header received
0924 04:16:11.973526 (+ 863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:11.973542 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:11.973870 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:11.974136 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:16:11.975004 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:11.976141 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:11.977132 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:11.977660 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:11.977778 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:11.980171 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:16:11.980178 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:16:11.980191 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:16:11.980217 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:16:11.982136 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:11.982509 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:11.982515 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:11.982519 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:11.982619 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:11.982877 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:11.982881 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:11.982883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:11.983269 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:16:11.983458 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:11.983706 (+ 248us) server_negotiation.cc:299] Negotiation successful
0924 04:16:11.983990 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":107,"threads_started":1}
I20260924 04:16:11.985994 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:11.976697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40549 (local address 127.0.0.1:52222)
0924 04:16:11.977055 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:11.977108 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 04:16:11.977221 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:16:11.977481 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:16:11.977487 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:16:11.977873 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:16:11.978036 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:11.978044 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:11.978994 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:11.978997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:11.979687 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:11.979697 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:11.980168 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:11.980172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:11.980274 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:11.980732 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:16:11.980746 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:16:11.982271 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:16:11.983731 (+ 1460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:11.983740 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:11.983746 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:11.984048 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:11.984285 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:11.984288 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:11.984290 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:11.984364 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:11.984707 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:16:11.984713 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:16:11.984824 (+ 111us) client_negotiation.cc:769] Sending connection context
0924 04:16:11.985649 (+ 825us) client_negotiation.cc:240] Negotiation successful
0924 04:16:11.985813 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":102,"threads_started":1}
I20260924 04:16:11.986022 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:11.976828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52222 (local address 127.0.98.62:40549)
0924 04:16:11.976946 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:11.976950 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:11.977170 (+ 220us) server_negotiation.cc:407] Connection header received
0924 04:16:11.977306 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:11.977309 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:11.977362 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:11.977425 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:16:11.978138 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:11.978861 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:11.979835 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:11.980036 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:11.980122 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:11.982394 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:16:11.982397 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:16:11.982399 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:16:11.982422 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:16:11.983636 (+ 1214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:11.984132 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:11.984135 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:11.984137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:11.984207 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:11.984436 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:11.984439 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:11.984441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:11.984615 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:16:11.984695 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:11.985762 (+ 1067us) server_negotiation.cc:299] Negotiation successful
0924 04:16:11.985868 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 04:16:11.986936 13936 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36767 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:16:11.989378 13784 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42381 } attempt: 3
I20260924 04:16:11.991524 13859 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42381 } attempt: 3
Sep 24 04:16:11 dist-test-slave-p401 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 24 04:16:11 dist-test-slave-p401 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 24 04:16:11 dist-test-slave-p401 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 04:16:11.996100 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:11.987472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58186 (local address 127.0.98.61:36767)
0924 04:16:11.987608 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:11.987612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:11.987625 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:16:11.987674 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:11.987677 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:11.987719 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:11.987777 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:16:11.988165 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:11.988730 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:11.989495 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:11.989824 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:11.990161 (+ 337us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:11.992966 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:16:11.992969 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:16:11.992972 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:16:11.992998 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:16:11.994434 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:11.994863 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:11.994866 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:11.994868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:11.994951 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:11.995254 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:11.995257 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:11.995259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:11.995463 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:16:11.995527 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:11.995835 (+ 308us) server_negotiation.cc:299] Negotiation successful
0924 04:16:11.995948 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 04:16:11.996546 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:11.987384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36767 (local address 127.0.0.1:58186)
0924 04:16:11.987534 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:11.987547 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:16:11.987607 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:16:11.987798 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:16:11.987800 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:16:11.987958 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:16:11.988056 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:11.988063 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:11.988836 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:11.988838 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:11.989388 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:11.989397 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:11.990301 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:11.990305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:11.990401 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:11.990814 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:16:11.990827 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:16:11.992812 (+ 1985us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:16:11.994546 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:11.994549 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:11.994551 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:11.994763 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:11.995051 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:11.995053 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:11.995055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:11.995138 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:11.995593 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:16:11.995596 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:16:11.995675 (+ 79us) client_negotiation.cc:769] Sending connection context
0924 04:16:11.996272 (+ 597us) client_negotiation.cc:240] Negotiation successful
0924 04:16:11.996404 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260924 04:16:11.996786 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:11.986960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52226 (local address 127.0.98.62:40549)
0924 04:16:11.987058 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:11.987062 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:11.987111 (+ 49us) server_negotiation.cc:407] Connection header received
0924 04:16:11.987251 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:11.987255 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:11.987299 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:11.987376 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:16:11.987916 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:11.988483 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:11.989262 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:11.989567 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:11.989651 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:11.992667 (+ 3016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:16:11.992670 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:16:11.992673 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:16:11.992703 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:16:11.994690 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:11.995112 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:11.995115 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:11.995117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:11.995190 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:11.995531 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:11.995534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:11.995536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:11.995730 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:16:11.995932 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:11.996537 (+ 605us) server_negotiation.cc:299] Negotiation successful
0924 04:16:11.996639 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34,"spinlock_wait_cycles":384}
I20260924 04:16:11.997126 13936 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42381 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 04:16:11 dist-test-slave-p401 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 04:16:11.998870 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:11.989683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42381 (local address 127.0.0.1:50204)
0924 04:16:11.989909 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:11.989922 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:16:11.989981 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:16:11.990173 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:16:11.990175 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:16:11.990343 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:16:11.990426 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:11.990432 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:11.991378 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:11.991380 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:11.991885 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:11.991893 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:11.992573 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:11.992577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:11.992659 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:11.993056 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:16:11.993070 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:16:11.994881 (+ 1811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:16:11.997421 (+ 2540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:11.997424 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:11.997427 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:11.997646 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:11.997931 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:11.997934 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:11.997936 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:11.998019 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:11.998425 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:16:11.998428 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:16:11.998512 (+ 84us) client_negotiation.cc:769] Sending connection context
0924 04:16:11.998602 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 04:16:11.998718 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"spinlock_wait_cycles":5248}
I20260924 04:16:11.999825 13939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:11.989905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50204 (local address 127.0.98.60:42381)
0924 04:16:11.989993 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:11.989997 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:11.990010 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:16:11.990038 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:11.990041 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:11.990078 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:11.990148 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:16:11.990551 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:11.991282 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:11.992183 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:11.992461 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:11.992557 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:11.995010 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:16:11.995013 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:16:11.995016 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:16:11.995037 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:16:11.997323 (+ 2286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:11.997741 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:11.997744 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:11.997746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:11.997823 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:11.998115 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:11.998118 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:11.998120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:11.998303 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:16:11.998385 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:11.999383 (+ 998us) server_negotiation.cc:299] Negotiation successful
0924 04:16:11.999681 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 04:16:12.002162 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:11.991909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42381 (local address 127.0.0.1:50212)
0924 04:16:11.992034 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:11.992047 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:16:11.992108 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:16:11.993622 (+ 1514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:16:11.993625 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:16:11.993806 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:16:11.993961 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:11.993968 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:11.994924 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:11.994928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:11.995760 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:11.995768 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:11.996334 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:11.996337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:11.996424 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:11.996853 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:16:11.996866 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:16:11.998536 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:16:12.000734 (+ 2198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:12.000737 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:12.000739 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:12.000969 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:12.001259 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:12.001262 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:12.001264 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:12.001341 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:12.001704 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:16:12.001707 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:16:12.001794 (+ 87us) client_negotiation.cc:769] Sending connection context
0924 04:16:12.001890 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 04:16:12.002009 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
Sep 24 04:16:11 dist-test-slave-p401 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 04:16:12.002703 13941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:11.992448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50212 (local address 127.0.98.60:42381)
0924 04:16:11.993410 (+ 962us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:11.993427 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:11.993442 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:16:11.993474 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:11.993476 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:11.993517 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:11.993601 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:16:11.994069 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:11.994810 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:11.995886 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:11.996214 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:11.996610 (+ 396us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:11.998658 (+ 2048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:16:11.998661 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:16:11.998664 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:16:11.998683 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:16:12.000631 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:12.001083 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:12.001086 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:12.001087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:12.001161 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:12.001437 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:12.001440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:12.001441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:12.001607 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:16:12.002396 (+ 789us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:12.002465 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 04:16:12.002561 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":104,"threads_started":1}
I20260924 04:16:12.003146 13784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0939eed012f423a9ad29b650876a140: Bootstrap starting.
Sep 24 04:16:12 dist-test-slave-p401 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 04:16:12.005992 13784 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0939eed012f423a9ad29b650876a140: Neither blocks nor log segments found. Creating new log.
I20260924 04:16:12.006347 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:11.997541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42381 (local address 127.0.0.1:50226)
0924 04:16:11.998054 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:11.998066 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:16:11.998130 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:16:11.998469 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:16:11.998472 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:16:11.998605 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:16:11.998675 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:11.998680 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:11.999611 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:11.999614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:12.000480 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:12.000489 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:12.001055 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:12.001058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:12.001127 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:12.001507 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:16:12.001520 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:16:12.003083 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:16:12.004919 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:12.004922 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:12.004924 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:12.005139 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:12.005406 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:12.005409 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:12.005411 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:12.005497 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:12.005867 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:16:12.005870 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:16:12.005957 (+ 87us) client_negotiation.cc:769] Sending connection context
0924 04:16:12.006060 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 04:16:12.006168 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"spinlock_wait_cycles":896}
I20260924 04:16:12.006399 13942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:11.997796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50226 (local address 127.0.98.60:42381)
0924 04:16:11.998087 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:11.998095 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:11.998105 (+ 10us) server_negotiation.cc:407] Connection header received
0924 04:16:11.998219 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:11.998221 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:11.998287 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:11.998366 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:16:11.998769 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:11.999524 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:12.000587 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:12.000894 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:12.000970 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:12.003216 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:16:12.003219 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:16:12.003222 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:16:12.003265 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:16:12.004811 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:12.005230 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:12.005233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:12.005235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:12.005314 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:12.005580 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:12.005583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:12.005585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:12.005767 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:16:12.005869 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:12.006128 (+ 259us) server_negotiation.cc:299] Negotiation successful
0924 04:16:12.006237 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":153,"threads_started":1}
I20260924 04:16:12.006861 13784 log.cc:824] T 00000000000000000000000000000000 P a0939eed012f423a9ad29b650876a140: Log is configured to *not* fsync() on all Append() calls
I20260924 04:16:12.008304 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:11.998829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58196 (local address 127.0.98.61:36767)
0924 04:16:11.998919 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:11.998922 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:11.999268 (+ 346us) server_negotiation.cc:407] Connection header received
0924 04:16:11.999357 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:11.999359 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:11.999397 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:11.999466 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:16:12.000019 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:12.000710 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:12.001923 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:12.002236 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:12.002328 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:12.004789 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:16:12.004793 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:16:12.004795 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:16:12.004818 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:16:12.006371 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:12.007092 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:12.007095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:12.007097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:12.007167 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:12.007454 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:12.007457 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:12.007459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:12.007657 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:16:12.007837 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:12.007970 (+ 133us) server_negotiation.cc:299] Negotiation successful
0924 04:16:12.008077 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 04:16:12.008992 13784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0939eed012f423a9ad29b650876a140: No bootstrap required, opened a new log
I20260924 04:16:12.010120 13936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6884677628724fb69810ce1816d18ab6: Bootstrap starting.
I20260924 04:16:12.011224 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:16:12.012809 13936 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6884677628724fb69810ce1816d18ab6: Neither blocks nor log segments found. Creating new log.
I20260924 04:16:12.013342 13784 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0939eed012f423a9ad29b650876a140 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0939eed012f423a9ad29b650876a140" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40549 } } peers { permanent_uuid: "739b14307d3e43199f304afbf39a29b3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36767 } } peers { permanent_uuid: "6884677628724fb69810ce1816d18ab6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42381 } }
I20260924 04:16:12.013535 13859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 739b14307d3e43199f304afbf39a29b3: Bootstrap starting.
I20260924 04:16:12.013571 13784 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0939eed012f423a9ad29b650876a140 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:16:12.013617 13784 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0939eed012f423a9ad29b650876a140 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0939eed012f423a9ad29b650876a140, State: Initialized, Role: FOLLOWER
I20260924 04:16:12.013625 13936 log.cc:824] T 00000000000000000000000000000000 P 6884677628724fb69810ce1816d18ab6: Log is configured to *not* fsync() on all Append() calls
I20260924 04:16:12.014108 13784 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0939eed012f423a9ad29b650876a140 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0939eed012f423a9ad29b650876a140" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40549 } } peers { permanent_uuid: "739b14307d3e43199f304afbf39a29b3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36767 } } peers { permanent_uuid: "6884677628724fb69810ce1816d18ab6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42381 } }
I20260924 04:16:12.014532 13949 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0939eed012f423a9ad29b650876a140 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0939eed012f423a9ad29b650876a140" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40549 } } peers { permanent_uuid: "739b14307d3e43199f304afbf39a29b3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36767 } } peers { permanent_uuid: "6884677628724fb69810ce1816d18ab6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42381 } } }
I20260924 04:16:12.014673 13949 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0939eed012f423a9ad29b650876a140 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:16:12.015416 13784 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0939eed012f423a9ad29b650876a140 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:16:12.015957 13936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6884677628724fb69810ce1816d18ab6: No bootstrap required, opened a new log
I20260924 04:16:12.016280 13859 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 739b14307d3e43199f304afbf39a29b3: Neither blocks nor log segments found. Creating new log.
I20260924 04:16:12.017141 13859 log.cc:824] T 00000000000000000000000000000000 P 739b14307d3e43199f304afbf39a29b3: Log is configured to *not* fsync() on all Append() calls
I20260924 04:16:12.019088 13859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 739b14307d3e43199f304afbf39a29b3: No bootstrap required, opened a new log
I20260924 04:16:12.020334 13936 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6884677628724fb69810ce1816d18ab6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0939eed012f423a9ad29b650876a140" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40549 } } peers { permanent_uuid: "739b14307d3e43199f304afbf39a29b3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36767 } } peers { permanent_uuid: "6884677628724fb69810ce1816d18ab6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42381 } }
I20260924 04:16:12.020517 13936 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6884677628724fb69810ce1816d18ab6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:16:12.020565 13936 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6884677628724fb69810ce1816d18ab6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6884677628724fb69810ce1816d18ab6, State: Initialized, Role: FOLLOWER
I20260924 04:16:12.020974 13936 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6884677628724fb69810ce1816d18ab6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0939eed012f423a9ad29b650876a140" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40549 } } peers { permanent_uuid: "739b14307d3e43199f304afbf39a29b3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36767 } } peers { permanent_uuid: "6884677628724fb69810ce1816d18ab6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42381 } }
I20260924 04:16:12.021391 13965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6884677628724fb69810ce1816d18ab6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0939eed012f423a9ad29b650876a140" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40549 } } peers { permanent_uuid: "739b14307d3e43199f304afbf39a29b3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36767 } } peers { permanent_uuid: "6884677628724fb69810ce1816d18ab6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42381 } } }
I20260924 04:16:12.021559 13965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6884677628724fb69810ce1816d18ab6 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:16:12.022130 13936 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6884677628724fb69810ce1816d18ab6 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 04:16:12.023082 13737 tablet.cc:2369] T 00000000000000000000000000000000 P a0939eed012f423a9ad29b650876a140: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:16:12.026729 13964 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a0939eed012f423a9ad29b650876a140: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:16:12.027125 13964 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260924 04:16:12.027288 13976 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6884677628724fb69810ce1816d18ab6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:16:12.027359 13976 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:16:12.024480 13716 master_runner.cc:186] Error getting master registration for 127.0.98.61:36767: OK, instance_id { permanent_uuid: "739b14307d3e43199f304afbf39a29b3" instance_seqno: 1790223371739740 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 04:16:12.029238 13716 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 04:16:12.029408 13859 raft_consensus.cc:348] T 00000000000000000000000000000000 P 739b14307d3e43199f304afbf39a29b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0939eed012f423a9ad29b650876a140" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40549 } } peers { permanent_uuid: "739b14307d3e43199f304afbf39a29b3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36767 } } peers { permanent_uuid: "6884677628724fb69810ce1816d18ab6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42381 } }
I20260924 04:16:12.029635 13859 raft_consensus.cc:374] T 00000000000000000000000000000000 P 739b14307d3e43199f304afbf39a29b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:16:12.029875 13859 raft_consensus.cc:729] T 00000000000000000000000000000000 P 739b14307d3e43199f304afbf39a29b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 739b14307d3e43199f304afbf39a29b3, State: Initialized, Role: FOLLOWER
I20260924 04:16:12.030440 13859 consensus_queue.cc:260] T 00000000000000000000000000000000 P 739b14307d3e43199f304afbf39a29b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0939eed012f423a9ad29b650876a140" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40549 } } peers { permanent_uuid: "739b14307d3e43199f304afbf39a29b3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36767 } } peers { permanent_uuid: "6884677628724fb69810ce1816d18ab6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42381 } }
I20260924 04:16:12.031347 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:12.015660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52232 (local address 127.0.98.62:40549)
0924 04:16:12.015778 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:12.015782 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:12.019280 (+ 3498us) server_negotiation.cc:407] Connection header received
0924 04:16:12.019479 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:12.019482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:12.019528 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:12.019576 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:16:12.020559 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:12.021305 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:12.022463 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:12.022827 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:12.022900 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:12.026343 (+ 3443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:16:12.026347 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:16:12.026351 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:16:12.026379 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:16:12.028556 (+ 2177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:12.029961 (+ 1405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:12.029964 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:12.029966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:12.030053 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:12.030354 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:12.030357 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:12.030359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:12.030562 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:16:12.030672 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:12.031031 (+ 359us) server_negotiation.cc:299] Negotiation successful
0924 04:16:12.031154 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 04:16:12.032840 13859 sys_catalog.cc:564] T 00000000000000000000000000000000 P 739b14307d3e43199f304afbf39a29b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:16:12.030059 13866 master_runner.cc:186] Error getting master registration for 127.0.98.61:36767: OK, instance_id { permanent_uuid: "739b14307d3e43199f304afbf39a29b3" instance_seqno: 1790223371739740 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 04:16:12.033263 13866 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 04:16:12.037319 13942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:12.016338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50240 (local address 127.0.98.60:42381)
0924 04:16:12.016450 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:12.016455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:12.024219 (+ 7764us) server_negotiation.cc:407] Connection header received
0924 04:16:12.025473 (+ 1254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:12.025477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:12.025524 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:12.025571 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:16:12.026766 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:12.027867 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:12.029925 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:12.030285 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:12.030387 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:12.033317 (+ 2930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:16:12.033332 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:16:12.033335 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:16:12.033360 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:16:12.034923 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:12.035356 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:12.035360 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:12.035362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:12.035438 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:12.035688 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:12.035691 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:12.035693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:12.036142 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:16:12.036287 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:12.037060 (+ 773us) server_negotiation.cc:299] Negotiation successful
0924 04:16:12.037170 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 04:16:12.039019 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:12.016025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58202 (local address 127.0.98.61:36767)
0924 04:16:12.017642 (+ 1617us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:12.017646 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:12.023523 (+ 5877us) server_negotiation.cc:407] Connection header received
0924 04:16:12.026986 (+ 3463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:12.026989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:12.027035 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:12.027103 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:16:12.031848 (+ 4745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:12.032526 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:12.033823 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:12.034121 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:12.034193 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:12.035686 (+ 1493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:16:12.035689 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:16:12.035692 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:16:12.035714 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:16:12.037132 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:12.037652 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:12.037655 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:12.037657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:12.037736 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:12.037994 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:12.037997 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:12.037999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:12.038190 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:16:12.038294 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:12.038833 (+ 539us) server_negotiation.cc:299] Negotiation successful
0924 04:16:12.038917 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1500}
I20260924 04:16:12.040474 13791 master_runner.cc:263] No leader master found from master 739b14307d3e43199f304afbf39a29b3
I20260924 04:16:12.040544 13791 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260924 04:16:12.041115 13990 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 739b14307d3e43199f304afbf39a29b3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:16:12.041190 13990 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:16:12.046253 13979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 739b14307d3e43199f304afbf39a29b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0939eed012f423a9ad29b650876a140" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40549 } } peers { permanent_uuid: "739b14307d3e43199f304afbf39a29b3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36767 } } peers { permanent_uuid: "6884677628724fb69810ce1816d18ab6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42381 } } }
I20260924 04:16:12.046396 13979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 739b14307d3e43199f304afbf39a29b3 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:16:12.096138 13965 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6884677628724fb69810ce1816d18ab6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:16:12.096342 13965 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6884677628724fb69810ce1816d18ab6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0939eed012f423a9ad29b650876a140" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40549 } } peers { permanent_uuid: "739b14307d3e43199f304afbf39a29b3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36767 } } peers { permanent_uuid: "6884677628724fb69810ce1816d18ab6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42381 } }
I20260924 04:16:12.097389 13965 leader_election.cc:288] T 00000000000000000000000000000000 P 6884677628724fb69810ce1816d18ab6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0939eed012f423a9ad29b650876a140 (127.0.98.62:40549), 739b14307d3e43199f304afbf39a29b3 (127.0.98.61:36767)
I20260924 04:16:12.098291 13758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6884677628724fb69810ce1816d18ab6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0939eed012f423a9ad29b650876a140" is_pre_election: true
I20260924 04:16:12.098613 13758 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a0939eed012f423a9ad29b650876a140 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6884677628724fb69810ce1816d18ab6 in term 0.
I20260924 04:16:12.099042 13833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6884677628724fb69810ce1816d18ab6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "739b14307d3e43199f304afbf39a29b3" is_pre_election: true
I20260924 04:16:12.099524 13885 leader_election.cc:302] T 00000000000000000000000000000000 P 6884677628724fb69810ce1816d18ab6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6884677628724fb69810ce1816d18ab6, a0939eed012f423a9ad29b650876a140; no voters:
I20260924 04:16:12.099958 13833 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 739b14307d3e43199f304afbf39a29b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6884677628724fb69810ce1816d18ab6 in term 0.
I20260924 04:16:12.100512 13965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6884677628724fb69810ce1816d18ab6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:16:12.100596 13965 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6884677628724fb69810ce1816d18ab6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:16:12.100646 13965 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6884677628724fb69810ce1816d18ab6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:16:12.101902 13965 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6884677628724fb69810ce1816d18ab6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0939eed012f423a9ad29b650876a140" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40549 } } peers { permanent_uuid: "739b14307d3e43199f304afbf39a29b3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36767 } } peers { permanent_uuid: "6884677628724fb69810ce1816d18ab6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42381 } }
I20260924 04:16:12.102234 13965 leader_election.cc:288] T 00000000000000000000000000000000 P 6884677628724fb69810ce1816d18ab6 [CANDIDATE]: Term 1 election: Requested vote from peers a0939eed012f423a9ad29b650876a140 (127.0.98.62:40549), 739b14307d3e43199f304afbf39a29b3 (127.0.98.61:36767)
I20260924 04:16:12.102806 13833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6884677628724fb69810ce1816d18ab6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "739b14307d3e43199f304afbf39a29b3"
I20260924 04:16:12.102922 13833 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 739b14307d3e43199f304afbf39a29b3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:16:12.104157 13833 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 739b14307d3e43199f304afbf39a29b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6884677628724fb69810ce1816d18ab6 in term 1.
I20260924 04:16:12.104496 13758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6884677628724fb69810ce1816d18ab6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0939eed012f423a9ad29b650876a140"
I20260924 04:16:12.104630 13758 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0939eed012f423a9ad29b650876a140 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:16:12.105932 13758 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a0939eed012f423a9ad29b650876a140 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6884677628724fb69810ce1816d18ab6 in term 1.
I20260924 04:16:12.106216 13885 leader_election.cc:302] T 00000000000000000000000000000000 P 6884677628724fb69810ce1816d18ab6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6884677628724fb69810ce1816d18ab6, a0939eed012f423a9ad29b650876a140; no voters:
I20260924 04:16:12.106482 13965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6884677628724fb69810ce1816d18ab6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:16:12.106693 13965 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6884677628724fb69810ce1816d18ab6 [term 1 LEADER]: Becoming Leader. State: Replica: 6884677628724fb69810ce1816d18ab6, State: Running, Role: LEADER
I20260924 04:16:12.107046 13965 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6884677628724fb69810ce1816d18ab6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0939eed012f423a9ad29b650876a140" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40549 } } peers { permanent_uuid: "739b14307d3e43199f304afbf39a29b3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36767 } } peers { permanent_uuid: "6884677628724fb69810ce1816d18ab6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42381 } }
I20260924 04:16:12.112025 13965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6884677628724fb69810ce1816d18ab6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6884677628724fb69810ce1816d18ab6. Latest consensus state: current_term: 1 leader_uuid: "6884677628724fb69810ce1816d18ab6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0939eed012f423a9ad29b650876a140" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40549 } } peers { permanent_uuid: "739b14307d3e43199f304afbf39a29b3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36767 } } peers { permanent_uuid: "6884677628724fb69810ce1816d18ab6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42381 } } }
I20260924 04:16:12.112144 13965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6884677628724fb69810ce1816d18ab6 [sys.catalog]: This master's current role is: LEADER
I20260924 04:16:12.112496 13994 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:16:12.113540 13994 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:16:12.120848 13833 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 739b14307d3e43199f304afbf39a29b3 [term 1 FOLLOWER]: Refusing update from remote peer 6884677628724fb69810ce1816d18ab6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:16:12.121477 13992 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6884677628724fb69810ce1816d18ab6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "739b14307d3e43199f304afbf39a29b3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:16:12.125802 13979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 739b14307d3e43199f304afbf39a29b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6884677628724fb69810ce1816d18ab6. Latest consensus state: current_term: 1 leader_uuid: "6884677628724fb69810ce1816d18ab6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0939eed012f423a9ad29b650876a140" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40549 } } peers { permanent_uuid: "739b14307d3e43199f304afbf39a29b3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36767 } } peers { permanent_uuid: "6884677628724fb69810ce1816d18ab6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42381 } } }
I20260924 04:16:12.125928 13979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 739b14307d3e43199f304afbf39a29b3 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:16:12.129004 13758 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a0939eed012f423a9ad29b650876a140 [term 1 FOLLOWER]: Refusing update from remote peer 6884677628724fb69810ce1816d18ab6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:16:12.129528 13992 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6884677628724fb69810ce1816d18ab6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0939eed012f423a9ad29b650876a140" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:16:12.133589 13949 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0939eed012f423a9ad29b650876a140 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6884677628724fb69810ce1816d18ab6. Latest consensus state: current_term: 1 leader_uuid: "6884677628724fb69810ce1816d18ab6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0939eed012f423a9ad29b650876a140" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40549 } } peers { permanent_uuid: "739b14307d3e43199f304afbf39a29b3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36767 } } peers { permanent_uuid: "6884677628724fb69810ce1816d18ab6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42381 } } }
I20260924 04:16:12.133703 13949 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0939eed012f423a9ad29b650876a140 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:16:12.140972 13965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6884677628724fb69810ce1816d18ab6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6884677628724fb69810ce1816d18ab6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0939eed012f423a9ad29b650876a140" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40549 } } peers { permanent_uuid: "739b14307d3e43199f304afbf39a29b3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36767 } } peers { permanent_uuid: "6884677628724fb69810ce1816d18ab6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42381 } } }
I20260924 04:16:12.141098 13965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6884677628724fb69810ce1816d18ab6 [sys.catalog]: This master's current role is: LEADER
I20260924 04:16:12.144487 13979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 739b14307d3e43199f304afbf39a29b3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6884677628724fb69810ce1816d18ab6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0939eed012f423a9ad29b650876a140" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40549 } } peers { permanent_uuid: "739b14307d3e43199f304afbf39a29b3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36767 } } peers { permanent_uuid: "6884677628724fb69810ce1816d18ab6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42381 } } }
I20260924 04:16:12.144601 13979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 739b14307d3e43199f304afbf39a29b3 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:16:12.144944 13995 mvcc.cc:205] Tried to move back new op lower bound from 7332754932199354368 to 7332754932162334720. Current Snapshot: MvccSnapshot[applied={T|T < 7332754932199354368}]
I20260924 04:16:12.145283 13996 mvcc.cc:205] Tried to move back new op lower bound from 7332754932199354368 to 7332754932162334720. Current Snapshot: MvccSnapshot[applied={T|T < 7332754932199354368}]
I20260924 04:16:12.145913 13992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6884677628724fb69810ce1816d18ab6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6884677628724fb69810ce1816d18ab6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0939eed012f423a9ad29b650876a140" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40549 } } peers { permanent_uuid: "739b14307d3e43199f304afbf39a29b3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36767 } } peers { permanent_uuid: "6884677628724fb69810ce1816d18ab6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42381 } } }
I20260924 04:16:12.146027 13992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6884677628724fb69810ce1816d18ab6 [sys.catalog]: This master's current role is: LEADER
I20260924 04:16:12.147011 13949 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0939eed012f423a9ad29b650876a140 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6884677628724fb69810ce1816d18ab6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0939eed012f423a9ad29b650876a140" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40549 } } peers { permanent_uuid: "739b14307d3e43199f304afbf39a29b3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36767 } } peers { permanent_uuid: "6884677628724fb69810ce1816d18ab6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42381 } } }
I20260924 04:16:12.147140 13949 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0939eed012f423a9ad29b650876a140 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:16:12.147783 13998 mvcc.cc:205] Tried to move back new op lower bound from 7332754932199354368 to 7332754932162334720. Current Snapshot: MvccSnapshot[applied={T|T < 7332754932199354368}]
I20260924 04:16:12.152050 13994 catalog_manager.cc:1347] Generated new cluster ID: 002571fbf31f4672a158d23f73a0de5b
I20260924 04:16:12.152112 13994 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:16:12.181149 13994 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:16:12.182428 13994 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:16:12.197139 13994 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6884677628724fb69810ce1816d18ab6: Generated new TSK 0
I20260924 04:16:12.197990 13994 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:16:13.028579 13964 catalog_manager.cc:1259] Loaded cluster ID: 002571fbf31f4672a158d23f73a0de5b
I20260924 04:16:13.028672 13964 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a0939eed012f423a9ad29b650876a140: loading cluster ID for follower catalog manager: success
I20260924 04:16:13.030409 13964 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a0939eed012f423a9ad29b650876a140: acquiring CA information for follower catalog manager: success
I20260924 04:16:13.031666 13964 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a0939eed012f423a9ad29b650876a140: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 04:16:13.042354 13990 catalog_manager.cc:1259] Loaded cluster ID: 002571fbf31f4672a158d23f73a0de5b
I20260924 04:16:13.042431 13990 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 739b14307d3e43199f304afbf39a29b3: loading cluster ID for follower catalog manager: success
I20260924 04:16:13.044055 13990 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 739b14307d3e43199f304afbf39a29b3: acquiring CA information for follower catalog manager: success
I20260924 04:16:13.045132 13990 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 739b14307d3e43199f304afbf39a29b3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 04:16:13.264770 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 13716
I20260924 04:16:13.270453 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 13791
I20260924 04:16:13.275079 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 13866
2026-09-24T04:16:13Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3019 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790223216265354-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:16:13 dist-test-slave-p401 krb5kdc[14008](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:16:13 dist-test-slave-p401 krb5kdc[14008](info): set up 2 sockets
Sep 24 04:16:13 dist-test-slave-p401 krb5kdc[14008](info): commencing operation
krb5kdc: starting...
W20260924 04:16:14.219135 392 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790223216265354-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790223216265354-392-0/krb5kdc/test-user.keytab.
Sep 24 04:16:14 dist-test-slave-p401 krb5kdc[14008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:16:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:16:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:16:14.334156 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790223216265354-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790223216265354-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790223216265354-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46247
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46247
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:16:14.444778 14021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:16:14.445076 14021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:16:14.445116 14021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:16:14.449087 14021 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:16:14.449146 14021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:16:14.449164 14021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:16:14.449179 14021 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:16:14.449193 14021 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:16:14.454553 14021 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790223216265354-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790223216265354-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46247
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790223216265354-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:16:14.455610 14021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:16:14.456579 14021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:16:14.464917 14026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:16:14.464960 14021 server_base.cc:1034] running on GCE node
W20260924 04:16:14.465040 14029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:16:14.464946 14027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:16:14.465626 14021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:16:14.466918 14021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:16:14.468113 14021 hybrid_clock.cc:648] HybridClock initialized: now 1790223374468082 us; error 48 us; skew 500 ppm
Sep 24 04:16:14 dist-test-slave-p401 krb5kdc[14008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223374, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:16:14.470892 14021 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:16:14.471834 14021 webserver.cc:460] Webserver started at http://127.0.98.62:33173/ using document root <none> and password file <none>
I20260924 04:16:14.472347 14021 fs_manager.cc:359] Metadata directory not provided
I20260924 04:16:14.472402 14021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:16:14.472563 14021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:16:14.474534 14021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790223216265354-392-0/minicluster-data/master-0/data/instance:
uuid: "053e27c1f55443fdb79e948df63f8367"
format_stamp: "Formatted at 2026-09-24 04:16:14 on dist-test-slave-p401"
server_key: "245e7f7bd0a088559428a63957c0c3a3"
server_key_iv: "736ab6ea24f2ff0b175329704f68a186"
server_key_version: "encryptionkey@0"
I20260924 04:16:14.475030 14021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790223216265354-392-0/minicluster-data/master-0/wal/instance:
uuid: "053e27c1f55443fdb79e948df63f8367"
format_stamp: "Formatted at 2026-09-24 04:16:14 on dist-test-slave-p401"
server_key: "245e7f7bd0a088559428a63957c0c3a3"
server_key_iv: "736ab6ea24f2ff0b175329704f68a186"
server_key_version: "encryptionkey@0"
I20260924 04:16:14.478572 14021 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 04:16:14.480921 14036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:16:14.482136 14021 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:16:14.482239 14021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790223216265354-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790223216265354-392-0/minicluster-data/master-0/wal
uuid: "053e27c1f55443fdb79e948df63f8367"
format_stamp: "Formatted at 2026-09-24 04:16:14 on dist-test-slave-p401"
server_key: "245e7f7bd0a088559428a63957c0c3a3"
server_key_iv: "736ab6ea24f2ff0b175329704f68a186"
server_key_version: "encryptionkey@0"
I20260924 04:16:14.482326 14021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790223216265354-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790223216265354-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790223216265354-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:16:14.503675 14021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:16:14.506845 14021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:16:14.507000 14021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:16:14.513610 14021 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46247
I20260924 04:16:14.513628 14088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46247 every 8 connection(s)
I20260924 04:16:14.514586 14021 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:16:14.517264 14089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:16:14.520354 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 14021
I20260924 04:16:14.520447 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790223216265354-392-0/minicluster-data/master-0/wal/instance
I20260924 04:16:14.520696 392 external_mini_cluster.cc:1428] Setting key 0e745551fa8aa27fbe028c137deae989
I20260924 04:16:14.523108 14089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 053e27c1f55443fdb79e948df63f8367: Bootstrap starting.
I20260924 04:16:14.525827 14089 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 053e27c1f55443fdb79e948df63f8367: Neither blocks nor log segments found. Creating new log.
I20260924 04:16:14.526719 14089 log.cc:824] T 00000000000000000000000000000000 P 053e27c1f55443fdb79e948df63f8367: Log is configured to *not* fsync() on all Append() calls
I20260924 04:16:14.529131 14089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 053e27c1f55443fdb79e948df63f8367: No bootstrap required, opened a new log
Sep 24 04:16:14 dist-test-slave-p401 krb5kdc[14008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:16:14.532936 14089 raft_consensus.cc:348] T 00000000000000000000000000000000 P 053e27c1f55443fdb79e948df63f8367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "053e27c1f55443fdb79e948df63f8367" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46247 } }
I20260924 04:16:14.533232 14089 raft_consensus.cc:374] T 00000000000000000000000000000000 P 053e27c1f55443fdb79e948df63f8367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:16:14.533289 14089 raft_consensus.cc:729] T 00000000000000000000000000000000 P 053e27c1f55443fdb79e948df63f8367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 053e27c1f55443fdb79e948df63f8367, State: Initialized, Role: FOLLOWER
I20260924 04:16:14.533806 14089 consensus_queue.cc:260] T 00000000000000000000000000000000 P 053e27c1f55443fdb79e948df63f8367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "053e27c1f55443fdb79e948df63f8367" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46247 } }
I20260924 04:16:14.533937 14089 raft_consensus.cc:388] T 00000000000000000000000000000000 P 053e27c1f55443fdb79e948df63f8367 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:16:14.533994 14089 raft_consensus.cc:482] T 00000000000000000000000000000000 P 053e27c1f55443fdb79e948df63f8367 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:16:14.534088 14089 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 053e27c1f55443fdb79e948df63f8367 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:16:14.535109 14089 raft_consensus.cc:504] T 00000000000000000000000000000000 P 053e27c1f55443fdb79e948df63f8367 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "053e27c1f55443fdb79e948df63f8367" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46247 } }
I20260924 04:16:14.535454 14089 leader_election.cc:302] T 00000000000000000000000000000000 P 053e27c1f55443fdb79e948df63f8367 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 053e27c1f55443fdb79e948df63f8367; no voters:
I20260924 04:16:14.535724 14089 leader_election.cc:288] T 00000000000000000000000000000000 P 053e27c1f55443fdb79e948df63f8367 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:16:14.535825 14094 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 053e27c1f55443fdb79e948df63f8367 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:16:14.535832 14092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:14.522006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52342 (local address 127.0.98.62:46247)
0924 04:16:14.522461 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:14.522481 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:14.522524 (+ 43us) server_negotiation.cc:407] Connection header received
0924 04:16:14.523339 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:14.523365 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:14.523769 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:14.524112 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:16:14.524899 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:14.526407 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:14.527606 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:14.528229 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:14.528352 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:14.531022 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:16:14.531026 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:16:14.531041 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:16:14.531067 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:16:14.532706 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:14.533252 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:14.533259 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:14.533265 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:14.533428 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:14.533712 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:14.533715 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:14.533716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:14.534225 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:16:14.534395 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:14.534648 (+ 253us) server_negotiation.cc:299] Negotiation successful
0924 04:16:14.534965 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":103,"threads_started":1}
I20260924 04:16:14.536054 14094 raft_consensus.cc:686] T 00000000000000000000000000000000 P 053e27c1f55443fdb79e948df63f8367 [term 1 LEADER]: Becoming Leader. State: Replica: 053e27c1f55443fdb79e948df63f8367, State: Running, Role: LEADER
I20260924 04:16:14.536361 14094 consensus_queue.cc:237] T 00000000000000000000000000000000 P 053e27c1f55443fdb79e948df63f8367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "053e27c1f55443fdb79e948df63f8367" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46247 } }
I20260924 04:16:14.536850 14089 sys_catalog.cc:564] T 00000000000000000000000000000000 P 053e27c1f55443fdb79e948df63f8367 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:16:14.537503 14095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 053e27c1f55443fdb79e948df63f8367 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "053e27c1f55443fdb79e948df63f8367" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "053e27c1f55443fdb79e948df63f8367" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46247 } } }
I20260924 04:16:14.537613 14095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 053e27c1f55443fdb79e948df63f8367 [sys.catalog]: This master's current role is: LEADER
I20260924 04:16:14.538136 14095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 053e27c1f55443fdb79e948df63f8367 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 053e27c1f55443fdb79e948df63f8367. Latest consensus state: current_term: 1 leader_uuid: "053e27c1f55443fdb79e948df63f8367" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "053e27c1f55443fdb79e948df63f8367" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46247 } } }
I20260924 04:16:14.538215 14095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 053e27c1f55443fdb79e948df63f8367 [sys.catalog]: This master's current role is: LEADER
I20260924 04:16:14.538488 14101 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:16:14.538923 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:16:14.541554 14101 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:16:14.546795 14101 catalog_manager.cc:1347] Generated new cluster ID: 992f5b193cf443ccbd1f0aa00293cf04
I20260924 04:16:14.546854 14101 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:16:14.559989 14101 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:16:14.560905 14101 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:16:14.569960 14101 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 053e27c1f55443fdb79e948df63f8367: Generated new TSK 0
I20260924 04:16:14.570650 14101 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:16:14.595098 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 14021
2026-09-24T04:16:14Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1319 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:16:14 dist-test-slave-p401 krb5kdc[14116](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:16:14 dist-test-slave-p401 krb5kdc[14116](info): set up 2 sockets
Sep 24 04:16:14 dist-test-slave-p401 krb5kdc[14116](info): commencing operation
krb5kdc: starting...
W20260924 04:16:15.567119 392 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/test-user.keytab.
Sep 24 04:16:15 dist-test-slave-p401 krb5kdc[14116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:16:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:16:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:16:15.680246 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43281
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43281
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:16:15.785252 14129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:16:15.785502 14129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:16:15.785540 14129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:16:15.789450 14129 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:16:15.789506 14129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:16:15.789525 14129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:16:15.789542 14129 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:16:15.789569 14129 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:16:15.794759 14129 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:16:15.795799 14129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:16:15.796768 14129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:16:15.804917 14135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:16:15.804917 14134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:16:15.805004 14129 server_base.cc:1034] running on GCE node
W20260924 04:16:15.804929 14137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:16:15.805689 14129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:16:15.806731 14129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:16:15.807894 14129 hybrid_clock.cc:648] HybridClock initialized: now 1790223375807881 us; error 29 us; skew 500 ppm
Sep 24 04:16:15 dist-test-slave-p401 krb5kdc[14116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223375, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:16:15.813220 14129 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:16:15.814127 14129 webserver.cc:460] Webserver started at http://127.0.98.62:34517/ using document root <none> and password file <none>
I20260924 04:16:15.814666 14129 fs_manager.cc:359] Metadata directory not provided
I20260924 04:16:15.814739 14129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:16:15.814950 14129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:16:15.817026 14129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/master-0/data/instance:
uuid: "ff323d589be24ca68b0f38f5703db99a"
format_stamp: "Formatted at 2026-09-24 04:16:15 on dist-test-slave-p401"
server_key: "f46dbda456d0b077c86cd9443b374b7e"
server_key_iv: "424d079b8077f4ffaa5b90beb0460532"
server_key_version: "encryptionkey@0"
I20260924 04:16:15.817571 14129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/master-0/wal/instance:
uuid: "ff323d589be24ca68b0f38f5703db99a"
format_stamp: "Formatted at 2026-09-24 04:16:15 on dist-test-slave-p401"
server_key: "f46dbda456d0b077c86cd9443b374b7e"
server_key_iv: "424d079b8077f4ffaa5b90beb0460532"
server_key_version: "encryptionkey@0"
I20260924 04:16:15.821228 14129 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:16:15.823552 14144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:16:15.824752 14129 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 04:16:15.824862 14129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/master-0/wal
uuid: "ff323d589be24ca68b0f38f5703db99a"
format_stamp: "Formatted at 2026-09-24 04:16:15 on dist-test-slave-p401"
server_key: "f46dbda456d0b077c86cd9443b374b7e"
server_key_iv: "424d079b8077f4ffaa5b90beb0460532"
server_key_version: "encryptionkey@0"
I20260924 04:16:15.824949 14129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:16:15.851558 14129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:16:15.854561 14129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:16:15.854732 14129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:16:15.861027 14129 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43281
I20260924 04:16:15.861264 14196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43281 every 8 connection(s)
I20260924 04:16:15.861928 14129 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:16:15.864564 14197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:16:15.865845 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 14129
I20260924 04:16:15.865967 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/master-0/wal/instance
I20260924 04:16:15.866245 392 external_mini_cluster.cc:1428] Setting key de47978e7cfa9a5de246f36e111d6154
I20260924 04:16:15.870812 14197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a: Bootstrap starting.
I20260924 04:16:15.873311 14197 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a: Neither blocks nor log segments found. Creating new log.
I20260924 04:16:15.873994 14197 log.cc:824] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a: Log is configured to *not* fsync() on all Append() calls
Sep 24 04:16:15 dist-test-slave-p401 krb5kdc[14116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:16:15.876029 14197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a: No bootstrap required, opened a new log
I20260924 04:16:15.880342 14197 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff323d589be24ca68b0f38f5703db99a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43281 } }
I20260924 04:16:15.880595 14197 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:16:15.880651 14197 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff323d589be24ca68b0f38f5703db99a, State: Initialized, Role: FOLLOWER
I20260924 04:16:15.880868 14200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:15.867605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33076 (local address 127.0.98.62:43281)
0924 04:16:15.867988 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:15.868006 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:15.868046 (+ 40us) server_negotiation.cc:407] Connection header received
0924 04:16:15.868799 (+ 753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:15.868820 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:15.869223 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:15.869567 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:16:15.870237 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:15.871701 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:15.872562 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:15.873228 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:15.873386 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:15.876041 (+ 2655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:16:15.876048 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:16:15.876064 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:16:15.876094 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:16:15.877860 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:15.878267 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:15.878274 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:15.878280 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:15.878432 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:15.878685 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:15.878688 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:15.878690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:15.879183 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:16:15.879407 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:15.879652 (+ 245us) server_negotiation.cc:299] Negotiation successful
0924 04:16:15.879945 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":97,"threads_started":1}
I20260924 04:16:15.881096 14197 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff323d589be24ca68b0f38f5703db99a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43281 } }
I20260924 04:16:15.881240 14197 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:16:15.881315 14197 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:16:15.881414 14197 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:16:15.882359 14197 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff323d589be24ca68b0f38f5703db99a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43281 } }
I20260924 04:16:15.882665 14197 leader_election.cc:302] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff323d589be24ca68b0f38f5703db99a; no voters:
I20260924 04:16:15.882966 14197 leader_election.cc:288] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:16:15.883090 14202 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:16:15.883318 14202 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a [term 1 LEADER]: Becoming Leader. State: Replica: ff323d589be24ca68b0f38f5703db99a, State: Running, Role: LEADER
I20260924 04:16:15.883605 14202 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff323d589be24ca68b0f38f5703db99a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43281 } }
I20260924 04:16:15.884059 14197 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:16:15.885191 14204 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff323d589be24ca68b0f38f5703db99a. Latest consensus state: current_term: 1 leader_uuid: "ff323d589be24ca68b0f38f5703db99a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff323d589be24ca68b0f38f5703db99a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43281 } } }
I20260924 04:16:15.885300 14204 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a [sys.catalog]: This master's current role is: LEADER
I20260924 04:16:15.885159 14203 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff323d589be24ca68b0f38f5703db99a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff323d589be24ca68b0f38f5703db99a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43281 } } }
I20260924 04:16:15.885667 14212 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:16:15.885692 14203 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a [sys.catalog]: This master's current role is: LEADER
I20260924 04:16:15.888307 14212 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:16:15.893126 14212 catalog_manager.cc:1347] Generated new cluster ID: ebd3bd98b35c4498a704f768bab163a6
I20260924 04:16:15.893193 14212 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:16:15.918124 14212 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:16:15.919441 14212 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:16:15.927531 14212 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a: Generated new TSK 0
I20260924 04:16:15.928231 14212 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:16:15.954392 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43281
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:16:16.060976 14223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:16:16.061204 14223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:16:16.061242 14223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:16:16.065097 14223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:16:16.065145 14223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:16:16.065198 14223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:16:16.070155 14223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:16:16.071043 14223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:16:16.072031 14223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:16:16.080998 14223 server_base.cc:1034] running on GCE node
W20260924 04:16:16.080964 14231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:16:16.080964 14228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:16:16.080972 14229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:16:16.081580 14223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:16:16.082149 14223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:16:16.083287 14223 hybrid_clock.cc:648] HybridClock initialized: now 1790223376083259 us; error 35 us; skew 500 ppm
Sep 24 04:16:16 dist-test-slave-p401 krb5kdc[14116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223376, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:16:16.085880 14223 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:16:16.086642 14223 webserver.cc:460] Webserver started at http://127.0.98.1:44461/ using document root <none> and password file <none>
I20260924 04:16:16.087126 14223 fs_manager.cc:359] Metadata directory not provided
I20260924 04:16:16.087178 14223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:16:16.087383 14223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:16:16.089264 14223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-0/data/instance:
uuid: "ca07c3fe6ef6478c97f25cfb38c349ac"
format_stamp: "Formatted at 2026-09-24 04:16:16 on dist-test-slave-p401"
server_key: "eb616c13f460039f1995842ba5d0fc85"
server_key_iv: "ba357bb10b7969a69c4c58212b96bd53"
server_key_version: "encryptionkey@0"
I20260924 04:16:16.089746 14223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ca07c3fe6ef6478c97f25cfb38c349ac"
format_stamp: "Formatted at 2026-09-24 04:16:16 on dist-test-slave-p401"
server_key: "eb616c13f460039f1995842ba5d0fc85"
server_key_iv: "ba357bb10b7969a69c4c58212b96bd53"
server_key_version: "encryptionkey@0"
I20260924 04:16:16.093247 14223 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:16:16.095466 14238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:16:16.096643 14223 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:16:16.096750 14223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-0/wal
uuid: "ca07c3fe6ef6478c97f25cfb38c349ac"
format_stamp: "Formatted at 2026-09-24 04:16:16 on dist-test-slave-p401"
server_key: "eb616c13f460039f1995842ba5d0fc85"
server_key_iv: "ba357bb10b7969a69c4c58212b96bd53"
server_key_version: "encryptionkey@0"
I20260924 04:16:16.096837 14223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:16:16.114143 14223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:16:16.117085 14223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:16:16.117259 14223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:16:16.117823 14223 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:16:16.118628 14223 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:16:16.118675 14223 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:16:16.118728 14223 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:16:16.118757 14223 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:16:16.127213 14223 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34429
I20260924 04:16:16.127295 14351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34429 every 8 connection(s)
I20260924 04:16:16.128139 14223 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:16:16.129994 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 14223
I20260924 04:16:16.130115 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:16:16.130401 392 external_mini_cluster.cc:1428] Setting key c14b4639de4a29b533bfae018ffad6af
Sep 24 04:16:16 dist-test-slave-p401 krb5kdc[14116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223376, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:16:16.141871 14200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:16.129829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42285 (local address 127.0.98.62:43281)
0924 04:16:16.129950 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:16.129953 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:16.130971 (+ 1018us) server_negotiation.cc:407] Connection header received
0924 04:16:16.132430 (+ 1459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:16.132433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:16.132481 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:16.132563 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:16:16.133801 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:16.134321 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:16.135484 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:16.135839 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:16.135951 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:16.138610 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:16:16.138614 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:16:16.138617 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:16:16.138647 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:16:16.140039 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:16.140527 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:16.140529 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:16.140530 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:16.140600 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:16.140858 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:16.140860 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:16.140861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:16.141019 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:16:16.141086 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:16.141588 (+ 502us) server_negotiation.cc:299] Negotiation successful
0924 04:16:16.141710 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 04:16:16.142818 14354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:16.130324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43281 (local address 127.0.98.1:42285)
0924 04:16:16.130812 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:16.130868 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 04:16:16.132231 (+ 1363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:16:16.132690 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:16:16.132696 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:16:16.133034 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:16:16.133658 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:16.133676 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:16.134433 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:16.134436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:16.135371 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:16.135380 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:16.136023 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:16.136027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:16.136183 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:16.136796 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:16:16.136814 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:16:16.138438 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:16:16.140154 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:16.140160 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:16.140175 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:16.140447 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:16.140669 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:16.140670 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:16.140672 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:16.140794 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:16.141097 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:16:16.141101 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:16:16.141347 (+ 246us) client_negotiation.cc:769] Sending connection context
0924 04:16:16.141531 (+ 184us) client_negotiation.cc:240] Negotiation successful
0924 04:16:16.141824 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":120,"threads_started":1}
I20260924 04:16:16.144035 14352 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43281
I20260924 04:16:16.144256 14352 heartbeater.cc:464] Registering TS with master...
I20260924 04:16:16.144795 14352 heartbeater.cc:511] Master 127.0.98.62:43281 requested a full tablet report, sending...
I20260924 04:16:16.146314 14161 ts_manager.cc:194] Registered new tserver with Master: ca07c3fe6ef6478c97f25cfb38c349ac (127.0.98.1:34429)
I20260924 04:16:16.147435 14161 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42285
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:16:16.156071 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43281
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:16:16.262269 14357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:16:16.262518 14357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:16:16.262578 14357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:16:16.266499 14357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:16:16.266557 14357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:16:16.266636 14357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:16:16.271803 14357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:16:16.272779 14357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:16:16.273742 14357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:16:16.282903 14365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:16:16.282912 14362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:16:16.282935 14363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:16:16.283272 14357 server_base.cc:1034] running on GCE node
I20260924 04:16:16.283733 14357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:16:16.284343 14357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:16:16.285495 14357 hybrid_clock.cc:648] HybridClock initialized: now 1790223376285476 us; error 30 us; skew 500 ppm
Sep 24 04:16:16 dist-test-slave-p401 krb5kdc[14116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223376, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:16:16.288062 14357 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:16:16.288820 14357 webserver.cc:460] Webserver started at http://127.0.98.2:36609/ using document root <none> and password file <none>
I20260924 04:16:16.289304 14357 fs_manager.cc:359] Metadata directory not provided
I20260924 04:16:16.289358 14357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:16:16.289517 14357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:16:16.291445 14357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-1/data/instance:
uuid: "683ba6100ecc4be5a17b03622dcad539"
format_stamp: "Formatted at 2026-09-24 04:16:16 on dist-test-slave-p401"
server_key: "7620d90c84aef57877cdca256c5f9472"
server_key_iv: "4185d02c74de83aa921fd676ac1eb203"
server_key_version: "encryptionkey@0"
I20260924 04:16:16.291949 14357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-1/wal/instance:
uuid: "683ba6100ecc4be5a17b03622dcad539"
format_stamp: "Formatted at 2026-09-24 04:16:16 on dist-test-slave-p401"
server_key: "7620d90c84aef57877cdca256c5f9472"
server_key_iv: "4185d02c74de83aa921fd676ac1eb203"
server_key_version: "encryptionkey@0"
I20260924 04:16:16.295326 14357 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:16:16.297518 14372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:16:16.298660 14357 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:16:16.298751 14357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-1/wal
uuid: "683ba6100ecc4be5a17b03622dcad539"
format_stamp: "Formatted at 2026-09-24 04:16:16 on dist-test-slave-p401"
server_key: "7620d90c84aef57877cdca256c5f9472"
server_key_iv: "4185d02c74de83aa921fd676ac1eb203"
server_key_version: "encryptionkey@0"
I20260924 04:16:16.298828 14357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:16:16.314584 14357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:16:16.317392 14357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:16:16.317533 14357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:16:16.318034 14357 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:16:16.318755 14357 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:16:16.318799 14357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:16:16.318845 14357 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:16:16.318866 14357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:16:16.326891 14357 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34529
I20260924 04:16:16.326934 14485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34529 every 8 connection(s)
I20260924 04:16:16.327744 14357 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:16:16.331743 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 14357
I20260924 04:16:16.331871 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:16:16.332103 392 external_mini_cluster.cc:1428] Setting key 5c0af326ae84df525de7e00f4675be58
Sep 24 04:16:16 dist-test-slave-p401 krb5kdc[14116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223376, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:16:16.341212 14200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:16.329440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59373 (local address 127.0.98.62:43281)
0924 04:16:16.329554 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:16.329557 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:16.330336 (+ 779us) server_negotiation.cc:407] Connection header received
0924 04:16:16.331610 (+ 1274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:16.331614 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:16.331669 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:16.331738 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:16:16.332971 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:16.333558 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:16.334643 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:16.334949 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:16.335047 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:16.337617 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:16:16.337622 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:16:16.337625 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:16:16.337658 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:16:16.339216 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:16.339697 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:16.339702 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:16.339704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:16.339803 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:16.340103 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:16.340107 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:16.340109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:16.340353 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:16:16.340464 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:16.340890 (+ 426us) server_negotiation.cc:299] Negotiation successful
0924 04:16:16.341021 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:16:16.341854 14488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:16.329799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43281 (local address 127.0.98.2:59373)
0924 04:16:16.330206 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:16.330246 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 04:16:16.331405 (+ 1159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:16:16.331924 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:16:16.331933 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:16:16.332300 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:16:16.332837 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:16.332851 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:16.333669 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:16.333671 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:16.334533 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:16.334542 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:16.335098 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:16.335102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:16.335301 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:16.335888 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:16:16.335904 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:16:16.337446 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:16:16.339355 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:16.339360 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:16.339371 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:16.339602 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:16.339893 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:16.339896 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:16.339898 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:16.340018 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:16.340445 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:16:16.340449 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:16:16.340659 (+ 210us) client_negotiation.cc:769] Sending connection context
0924 04:16:16.340805 (+ 146us) client_negotiation.cc:240] Negotiation successful
0924 04:16:16.341046 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":96,"threads_started":1}
I20260924 04:16:16.343007 14486 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43281
I20260924 04:16:16.343204 14486 heartbeater.cc:464] Registering TS with master...
I20260924 04:16:16.343670 14486 heartbeater.cc:511] Master 127.0.98.62:43281 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 04:16:16.344717 14161 ts_manager.cc:194] Registered new tserver with Master: 683ba6100ecc4be5a17b03622dcad539 (127.0.98.2:34529)
I20260924 04:16:16.345444 14161 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59373
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:16:16.359861 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43281
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:16:16.466714 14491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:16:16.467047 14491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:16:16.467108 14491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:16:16.471071 14491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:16:16.471129 14491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:16:16.471206 14491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:16:16.476390 14491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:16:16.477388 14491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:16:16.478381 14491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:16:16.487460 14496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:16:16.487493 14491 server_base.cc:1034] running on GCE node
W20260924 04:16:16.487550 14497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:16:16.487464 14499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:16:16.488036 14491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:16:16.488612 14491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:16:16.489761 14491 hybrid_clock.cc:648] HybridClock initialized: now 1790223376489757 us; error 43 us; skew 500 ppm
Sep 24 04:16:16 dist-test-slave-p401 krb5kdc[14116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223376, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:16:16.492364 14491 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:16:16.493148 14491 webserver.cc:460] Webserver started at http://127.0.98.3:39025/ using document root <none> and password file <none>
I20260924 04:16:16.493643 14491 fs_manager.cc:359] Metadata directory not provided
I20260924 04:16:16.493705 14491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:16:16.493880 14491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:16:16.495800 14491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-2/data/instance:
uuid: "3f4ec3e52e1a494c82be7558bea4ff26"
format_stamp: "Formatted at 2026-09-24 04:16:16 on dist-test-slave-p401"
server_key: "53ede7cc0860d0b18f63e3bf39c5ab03"
server_key_iv: "afc2b0656bde7f3c45894ad4f3608f74"
server_key_version: "encryptionkey@0"
I20260924 04:16:16.496299 14491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3f4ec3e52e1a494c82be7558bea4ff26"
format_stamp: "Formatted at 2026-09-24 04:16:16 on dist-test-slave-p401"
server_key: "53ede7cc0860d0b18f63e3bf39c5ab03"
server_key_iv: "afc2b0656bde7f3c45894ad4f3608f74"
server_key_version: "encryptionkey@0"
I20260924 04:16:16.499678 14491 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 04:16:16.501935 14506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:16:16.503072 14491 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:16:16.503181 14491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-2/wal
uuid: "3f4ec3e52e1a494c82be7558bea4ff26"
format_stamp: "Formatted at 2026-09-24 04:16:16 on dist-test-slave-p401"
server_key: "53ede7cc0860d0b18f63e3bf39c5ab03"
server_key_iv: "afc2b0656bde7f3c45894ad4f3608f74"
server_key_version: "encryptionkey@0"
I20260924 04:16:16.503306 14491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:16:16.517715 14491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:16:16.520823 14491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:16:16.520993 14491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:16:16.521564 14491 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:16:16.522369 14491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:16:16.522415 14491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:16:16.522466 14491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:16:16.522495 14491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:16:16.531077 14491 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36613
I20260924 04:16:16.531148 14619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36613 every 8 connection(s)
I20260924 04:16:16.532063 14491 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:16:16.535316 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 14491
I20260924 04:16:16.535401 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:16:16.535638 392 external_mini_cluster.cc:1428] Setting key 79c7cde6224afa9ba549c99513ef8129
Sep 24 04:16:16 dist-test-slave-p401 krb5kdc[14116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223376, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:16:16.545032 14200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:16.533967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40731 (local address 127.0.98.62:43281)
0924 04:16:16.534130 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:16.534135 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:16.535039 (+ 904us) server_negotiation.cc:407] Connection header received
0924 04:16:16.536418 (+ 1379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:16.536420 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:16.536472 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:16.536547 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:16:16.537862 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:16.538388 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:16.539520 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:16.539740 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:16.539833 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:16.542028 (+ 2195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:16:16.542032 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:16:16.542035 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:16:16.542060 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:16:16.543201 (+ 1141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:16.543673 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:16.543677 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:16.543678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:16.543751 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:16.544046 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:16.544049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:16.544051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:16.544214 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:16:16.544295 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:16.544749 (+ 454us) server_negotiation.cc:299] Negotiation successful
0924 04:16:16.544871 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 04:16:16.545746 14622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:16.534432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43281 (local address 127.0.98.3:40731)
0924 04:16:16.534902 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:16.534942 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 04:16:16.536204 (+ 1262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:16:16.536706 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:16:16.536712 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:16:16.537179 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:16:16.537732 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:16.537746 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:16.538497 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:16.538501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:16.539416 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:16.539426 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:16.539873 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:16.539877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:16.540042 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:16.540497 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:16:16.540511 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:16:16.541790 (+ 1279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:16:16.543337 (+ 1547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:16.543342 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:16.543356 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:16.543584 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:16.543836 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:16.543839 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:16.543841 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:16.543963 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:16.544311 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:16:16.544317 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:16:16.544537 (+ 220us) client_negotiation.cc:769] Sending connection context
0924 04:16:16.544691 (+ 154us) client_negotiation.cc:240] Negotiation successful
0924 04:16:16.544989 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":124,"threads_started":1}
I20260924 04:16:16.546703 14620 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43281
I20260924 04:16:16.546906 14620 heartbeater.cc:464] Registering TS with master...
I20260924 04:16:16.547384 14620 heartbeater.cc:511] Master 127.0.98.62:43281 requested a full tablet report, sending...
I20260924 04:16:16.548413 14161 ts_manager.cc:194] Registered new tserver with Master: 3f4ec3e52e1a494c82be7558bea4ff26 (127.0.98.3:36613)
I20260924 04:16:16.548991 14161 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40731
I20260924 04:16:16.549247 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:16:16.557380 14200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:16.551041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33080 (local address 127.0.98.62:43281)
0924 04:16:16.551142 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:16.551146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:16.551306 (+ 160us) server_negotiation.cc:407] Connection header received
0924 04:16:16.551420 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:16.551422 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:16.551467 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:16.551528 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:16:16.552114 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:16.552602 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:16.553352 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:16.553579 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:16.553674 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:16.554785 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:16:16.554788 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:16:16.554790 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:16:16.554812 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:16:16.556100 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:16.556457 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:16.556460 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:16.556461 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:16.556525 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:16.556710 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:16.556713 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:16.556714 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:16.556940 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:16:16.557036 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:16.557183 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 04:16:16.557269 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 04:16:16.559932 14161 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33080:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:16:16.562834 14161 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:16:16.576539 14631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:16.571273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34429 (local address 127.0.0.1:39802)
0924 04:16:16.571796 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:16.571808 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:16:16.571882 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:16:16.572349 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:16:16.572355 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:16:16.572395 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:16:16.572578 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:16.572585 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:16.573682 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:16.573685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:16.574933 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:16.574943 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:16.575970 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:16.575974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:16.576080 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:16.576123 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 04:16:16.576240 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 04:16:16.576366 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"mutex_wait_us":72,"spinlock_wait_cycles":10240,"thread_start_us":79,"threads_started":1}
I20260924 04:16:16.576700 14634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:16.570642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36613 (local address 127.0.0.1:35094)
0924 04:16:16.571548 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:16.571559 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:16:16.571620 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:16:16.572518 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:16:16.572521 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:16:16.572535 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:16:16.572630 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:16.572635 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:16.574020 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:16.574023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:16.575283 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:16.575294 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:16.576258 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:16.576262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:16.576402 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:16.576424 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 04:16:16.576491 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 04:16:16.576566 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":852,"mutex_wait_us":71,"thread_start_us":116,"threads_started":1}
I20260924 04:16:16.576741 14635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:16.571356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35094 (local address 127.0.98.3:36613)
0924 04:16:16.572123 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:16.572133 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:16.572152 (+ 19us) server_negotiation.cc:407] Connection header received
0924 04:16:16.572219 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:16.572223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:16.572385 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:16.572531 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:16:16.572709 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:16.573887 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:16.575399 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:16.576150 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:16.576246 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:16.576380 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:16.576529 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 04:16:16.576622 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":167,"threads_started":1}
I20260924 04:16:16.576995 14633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:16.570677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34424 (local address 127.0.98.2:34529)
0924 04:16:16.570984 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:16.570995 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:16.571721 (+ 726us) server_negotiation.cc:407] Connection header received
0924 04:16:16.571800 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:16.571806 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:16.572017 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:16.572161 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:16:16.572695 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:16.573881 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:16.575423 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:16.575997 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:16.576092 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:16.576223 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:16.576773 (+ 550us) server_negotiation.cc:299] Negotiation successful
0924 04:16:16.576872 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":99,"threads_started":1}
I20260924 04:16:16.577224 14632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:16.570537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34529 (local address 127.0.0.1:34424)
0924 04:16:16.571397 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:16.571446 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 04:16:16.571592 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:16:16.572353 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:16:16.572356 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:16:16.572392 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:16:16.572566 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:16.572576 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:16.574168 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:16.574171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:16.575311 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:16.575318 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:16.576281 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:16.576284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:16.576429 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:16.576451 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 04:16:16.576514 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 04:16:16.576578 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"mutex_wait_us":63,"spinlock_wait_cycles":3328,"thread_start_us":69,"threads_started":1}
I20260924 04:16:16.577596 14636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:16.571473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39802 (local address 127.0.98.1:34429)
0924 04:16:16.571827 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:16.571839 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:16.571864 (+ 25us) server_negotiation.cc:407] Connection header received
0924 04:16:16.571983 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:16.571989 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:16.572173 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:16.572715 (+ 542us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:16:16.572861 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:16.573565 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:16.575056 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:16.575828 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:16.575950 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:16.576100 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:16.577375 (+ 1275us) server_negotiation.cc:299] Negotiation successful
0924 04:16:16.577477 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":95,"threads_started":1}
I20260924 04:16:16.579509 14554 tablet_service.cc:1467] Processing CreateTablet for tablet af429b0b011b406d98bf154b004313dc (DEFAULT_TABLE table=test-table [id=89fcc549a36945bf8a42ce746731cd7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:16:16.580432 14554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af429b0b011b406d98bf154b004313dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:16:16.580257 14286 tablet_service.cc:1467] Processing CreateTablet for tablet af429b0b011b406d98bf154b004313dc (DEFAULT_TABLE table=test-table [id=89fcc549a36945bf8a42ce746731cd7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:16:16.580256 14420 tablet_service.cc:1467] Processing CreateTablet for tablet af429b0b011b406d98bf154b004313dc (DEFAULT_TABLE table=test-table [id=89fcc549a36945bf8a42ce746731cd7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:16:16.581264 14286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af429b0b011b406d98bf154b004313dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:16:16.581264 14420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af429b0b011b406d98bf154b004313dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:16:16.587319 14637 tablet_bootstrap.cc:492] T af429b0b011b406d98bf154b004313dc P 3f4ec3e52e1a494c82be7558bea4ff26: Bootstrap starting.
I20260924 04:16:16.588348 14638 tablet_bootstrap.cc:492] T af429b0b011b406d98bf154b004313dc P ca07c3fe6ef6478c97f25cfb38c349ac: Bootstrap starting.
I20260924 04:16:16.588750 14639 tablet_bootstrap.cc:492] T af429b0b011b406d98bf154b004313dc P 683ba6100ecc4be5a17b03622dcad539: Bootstrap starting.
I20260924 04:16:16.590291 14637 tablet_bootstrap.cc:654] T af429b0b011b406d98bf154b004313dc P 3f4ec3e52e1a494c82be7558bea4ff26: Neither blocks nor log segments found. Creating new log.
I20260924 04:16:16.591179 14639 tablet_bootstrap.cc:654] T af429b0b011b406d98bf154b004313dc P 683ba6100ecc4be5a17b03622dcad539: Neither blocks nor log segments found. Creating new log.
I20260924 04:16:16.591219 14637 log.cc:824] T af429b0b011b406d98bf154b004313dc P 3f4ec3e52e1a494c82be7558bea4ff26: Log is configured to *not* fsync() on all Append() calls
I20260924 04:16:16.592029 14639 log.cc:824] T af429b0b011b406d98bf154b004313dc P 683ba6100ecc4be5a17b03622dcad539: Log is configured to *not* fsync() on all Append() calls
I20260924 04:16:16.592588 14638 tablet_bootstrap.cc:654] T af429b0b011b406d98bf154b004313dc P ca07c3fe6ef6478c97f25cfb38c349ac: Neither blocks nor log segments found. Creating new log.
I20260924 04:16:16.593276 14637 tablet_bootstrap.cc:492] T af429b0b011b406d98bf154b004313dc P 3f4ec3e52e1a494c82be7558bea4ff26: No bootstrap required, opened a new log
I20260924 04:16:16.593355 14638 log.cc:824] T af429b0b011b406d98bf154b004313dc P ca07c3fe6ef6478c97f25cfb38c349ac: Log is configured to *not* fsync() on all Append() calls
I20260924 04:16:16.593465 14637 ts_tablet_manager.cc:1397] T af429b0b011b406d98bf154b004313dc P 3f4ec3e52e1a494c82be7558bea4ff26: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260924 04:16:16.593953 14639 tablet_bootstrap.cc:492] T af429b0b011b406d98bf154b004313dc P 683ba6100ecc4be5a17b03622dcad539: No bootstrap required, opened a new log
I20260924 04:16:16.594116 14639 ts_tablet_manager.cc:1397] T af429b0b011b406d98bf154b004313dc P 683ba6100ecc4be5a17b03622dcad539: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260924 04:16:16.595171 14638 tablet_bootstrap.cc:492] T af429b0b011b406d98bf154b004313dc P ca07c3fe6ef6478c97f25cfb38c349ac: No bootstrap required, opened a new log
I20260924 04:16:16.595377 14638 ts_tablet_manager.cc:1397] T af429b0b011b406d98bf154b004313dc P ca07c3fe6ef6478c97f25cfb38c349ac: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 04:16:16.596913 14639 raft_consensus.cc:348] T af429b0b011b406d98bf154b004313dc P 683ba6100ecc4be5a17b03622dcad539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "683ba6100ecc4be5a17b03622dcad539" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 } } peers { permanent_uuid: "3f4ec3e52e1a494c82be7558bea4ff26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36613 } } peers { permanent_uuid: "ca07c3fe6ef6478c97f25cfb38c349ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34429 } }
I20260924 04:16:16.597189 14639 raft_consensus.cc:374] T af429b0b011b406d98bf154b004313dc P 683ba6100ecc4be5a17b03622dcad539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:16:16.597273 14639 raft_consensus.cc:729] T af429b0b011b406d98bf154b004313dc P 683ba6100ecc4be5a17b03622dcad539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 683ba6100ecc4be5a17b03622dcad539, State: Initialized, Role: FOLLOWER
I20260924 04:16:16.597285 14637 raft_consensus.cc:348] T af429b0b011b406d98bf154b004313dc P 3f4ec3e52e1a494c82be7558bea4ff26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "683ba6100ecc4be5a17b03622dcad539" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 } } peers { permanent_uuid: "3f4ec3e52e1a494c82be7558bea4ff26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36613 } } peers { permanent_uuid: "ca07c3fe6ef6478c97f25cfb38c349ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34429 } }
I20260924 04:16:16.597585 14637 raft_consensus.cc:374] T af429b0b011b406d98bf154b004313dc P 3f4ec3e52e1a494c82be7558bea4ff26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:16:16.597640 14637 raft_consensus.cc:729] T af429b0b011b406d98bf154b004313dc P 3f4ec3e52e1a494c82be7558bea4ff26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f4ec3e52e1a494c82be7558bea4ff26, State: Initialized, Role: FOLLOWER
I20260924 04:16:16.597720 14639 consensus_queue.cc:260] T af429b0b011b406d98bf154b004313dc P 683ba6100ecc4be5a17b03622dcad539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "683ba6100ecc4be5a17b03622dcad539" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 } } peers { permanent_uuid: "3f4ec3e52e1a494c82be7558bea4ff26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36613 } } peers { permanent_uuid: "ca07c3fe6ef6478c97f25cfb38c349ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34429 } }
I20260924 04:16:16.598052 14637 consensus_queue.cc:260] T af429b0b011b406d98bf154b004313dc P 3f4ec3e52e1a494c82be7558bea4ff26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "683ba6100ecc4be5a17b03622dcad539" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 } } peers { permanent_uuid: "3f4ec3e52e1a494c82be7558bea4ff26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36613 } } peers { permanent_uuid: "ca07c3fe6ef6478c97f25cfb38c349ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34429 } }
I20260924 04:16:16.598297 14486 heartbeater.cc:503] Master 127.0.98.62:43281 was elected leader, sending a full tablet report...
I20260924 04:16:16.599056 14637 ts_tablet_manager.cc:1428] T af429b0b011b406d98bf154b004313dc P 3f4ec3e52e1a494c82be7558bea4ff26: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260924 04:16:16.598940 14638 raft_consensus.cc:348] T af429b0b011b406d98bf154b004313dc P ca07c3fe6ef6478c97f25cfb38c349ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "683ba6100ecc4be5a17b03622dcad539" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 } } peers { permanent_uuid: "3f4ec3e52e1a494c82be7558bea4ff26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36613 } } peers { permanent_uuid: "ca07c3fe6ef6478c97f25cfb38c349ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34429 } }
I20260924 04:16:16.599156 14638 raft_consensus.cc:374] T af429b0b011b406d98bf154b004313dc P ca07c3fe6ef6478c97f25cfb38c349ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:16:16.599203 14638 raft_consensus.cc:729] T af429b0b011b406d98bf154b004313dc P ca07c3fe6ef6478c97f25cfb38c349ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca07c3fe6ef6478c97f25cfb38c349ac, State: Initialized, Role: FOLLOWER
I20260924 04:16:16.599395 14620 heartbeater.cc:503] Master 127.0.98.62:43281 was elected leader, sending a full tablet report...
I20260924 04:16:16.599570 14639 ts_tablet_manager.cc:1428] T af429b0b011b406d98bf154b004313dc P 683ba6100ecc4be5a17b03622dcad539: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260924 04:16:16.599659 14638 consensus_queue.cc:260] T af429b0b011b406d98bf154b004313dc P ca07c3fe6ef6478c97f25cfb38c349ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "683ba6100ecc4be5a17b03622dcad539" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 } } peers { permanent_uuid: "3f4ec3e52e1a494c82be7558bea4ff26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36613 } } peers { permanent_uuid: "ca07c3fe6ef6478c97f25cfb38c349ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34429 } }
I20260924 04:16:16.600219 14352 heartbeater.cc:503] Master 127.0.98.62:43281 was elected leader, sending a full tablet report...
I20260924 04:16:16.601138 14638 ts_tablet_manager.cc:1428] T af429b0b011b406d98bf154b004313dc P ca07c3fe6ef6478c97f25cfb38c349ac: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260924 04:16:16.629236 14353 tablet.cc:2369] T af429b0b011b406d98bf154b004313dc P ca07c3fe6ef6478c97f25cfb38c349ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:16:16.662690 14644 raft_consensus.cc:482] T af429b0b011b406d98bf154b004313dc P 3f4ec3e52e1a494c82be7558bea4ff26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:16:16.662905 14644 raft_consensus.cc:504] T af429b0b011b406d98bf154b004313dc P 3f4ec3e52e1a494c82be7558bea4ff26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "683ba6100ecc4be5a17b03622dcad539" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 } } peers { permanent_uuid: "3f4ec3e52e1a494c82be7558bea4ff26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36613 } } peers { permanent_uuid: "ca07c3fe6ef6478c97f25cfb38c349ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34429 } }
I20260924 04:16:16.664068 14644 leader_election.cc:288] T af429b0b011b406d98bf154b004313dc P 3f4ec3e52e1a494c82be7558bea4ff26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 683ba6100ecc4be5a17b03622dcad539 (127.0.98.2:34529), ca07c3fe6ef6478c97f25cfb38c349ac (127.0.98.1:34429)
I20260924 04:16:16.667675 14633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:16.664421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39821 (local address 127.0.98.2:34529)
0924 04:16:16.664560 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:16.664565 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:16.664582 (+ 17us) server_negotiation.cc:407] Connection header received
0924 04:16:16.664711 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:16.664714 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:16.664761 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:16.664840 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:16:16.665130 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:16.665721 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:16.666746 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:16.667140 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:16.667260 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:16.667324 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:16.667472 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 04:16:16.667547 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 04:16:16.667677 14622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:16.664299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34529 (local address 127.0.98.3:39821)
0924 04:16:16.664484 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:16.664499 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:16:16.664590 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:16:16.664868 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:16:16.664871 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:16:16.664896 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:16:16.665019 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:16.665026 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:16.665821 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:16.665824 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:16.666626 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:16.666635 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:16.667260 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:16.667264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:16.667366 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:16.667383 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 04:16:16.667445 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 04:16:16.667558 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":34,"spinlock_wait_cycles":5376}
I20260924 04:16:16.667788 14636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:16.664637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32889 (local address 127.0.98.1:34429)
0924 04:16:16.664760 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:16.664764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:16.664824 (+ 60us) server_negotiation.cc:407] Connection header received
0924 04:16:16.665047 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:16.665050 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:16.665096 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:16.665175 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:16:16.665419 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:16.665895 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:16.666865 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:16.667288 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:16.667378 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:16.667434 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:16.667609 (+ 175us) server_negotiation.cc:299] Negotiation successful
0924 04:16:16.667673 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 04:16:16.667829 14646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:16.664344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34429 (local address 127.0.98.3:32889)
0924 04:16:16.664744 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:16.664757 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:16:16.664894 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:16:16.665187 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:16:16.665190 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:16:16.665207 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:16:16.665305 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:16.665311 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:16.666003 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:16.666007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:16.666748 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:16.666756 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:16.667390 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:16.667393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:16.667506 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:16.667522 (+ 16us) client_negotiation.cc:769] Sending connection context
0924 04:16:16.667583 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 04:16:16.667647 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":1,"spinlock_wait_cycles":384,"thread_start_us":128,"threads_started":1}
I20260924 04:16:16.668383 14306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af429b0b011b406d98bf154b004313dc" candidate_uuid: "3f4ec3e52e1a494c82be7558bea4ff26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca07c3fe6ef6478c97f25cfb38c349ac" is_pre_election: true
I20260924 04:16:16.668399 14440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af429b0b011b406d98bf154b004313dc" candidate_uuid: "3f4ec3e52e1a494c82be7558bea4ff26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "683ba6100ecc4be5a17b03622dcad539" is_pre_election: true
I20260924 04:16:16.668649 14306 raft_consensus.cc:2454] T af429b0b011b406d98bf154b004313dc P ca07c3fe6ef6478c97f25cfb38c349ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f4ec3e52e1a494c82be7558bea4ff26 in term 0.
I20260924 04:16:16.668658 14440 raft_consensus.cc:2454] T af429b0b011b406d98bf154b004313dc P 683ba6100ecc4be5a17b03622dcad539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f4ec3e52e1a494c82be7558bea4ff26 in term 0.
I20260924 04:16:16.669044 14507 leader_election.cc:302] T af429b0b011b406d98bf154b004313dc P 3f4ec3e52e1a494c82be7558bea4ff26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f4ec3e52e1a494c82be7558bea4ff26, ca07c3fe6ef6478c97f25cfb38c349ac; no voters:
I20260924 04:16:16.669297 14644 raft_consensus.cc:2781] T af429b0b011b406d98bf154b004313dc P 3f4ec3e52e1a494c82be7558bea4ff26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:16:16.669385 14644 raft_consensus.cc:482] T af429b0b011b406d98bf154b004313dc P 3f4ec3e52e1a494c82be7558bea4ff26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:16:16.669436 14644 raft_consensus.cc:3037] T af429b0b011b406d98bf154b004313dc P 3f4ec3e52e1a494c82be7558bea4ff26 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:16:16.670514 14644 raft_consensus.cc:504] T af429b0b011b406d98bf154b004313dc P 3f4ec3e52e1a494c82be7558bea4ff26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "683ba6100ecc4be5a17b03622dcad539" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 } } peers { permanent_uuid: "3f4ec3e52e1a494c82be7558bea4ff26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36613 } } peers { permanent_uuid: "ca07c3fe6ef6478c97f25cfb38c349ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34429 } }
I20260924 04:16:16.670867 14644 leader_election.cc:288] T af429b0b011b406d98bf154b004313dc P 3f4ec3e52e1a494c82be7558bea4ff26 [CANDIDATE]: Term 1 election: Requested vote from peers 683ba6100ecc4be5a17b03622dcad539 (127.0.98.2:34529), ca07c3fe6ef6478c97f25cfb38c349ac (127.0.98.1:34429)
I20260924 04:16:16.671166 14440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af429b0b011b406d98bf154b004313dc" candidate_uuid: "3f4ec3e52e1a494c82be7558bea4ff26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "683ba6100ecc4be5a17b03622dcad539"
I20260924 04:16:16.671208 14306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af429b0b011b406d98bf154b004313dc" candidate_uuid: "3f4ec3e52e1a494c82be7558bea4ff26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca07c3fe6ef6478c97f25cfb38c349ac"
I20260924 04:16:16.671293 14440 raft_consensus.cc:3037] T af429b0b011b406d98bf154b004313dc P 683ba6100ecc4be5a17b03622dcad539 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:16:16.671355 14306 raft_consensus.cc:3037] T af429b0b011b406d98bf154b004313dc P ca07c3fe6ef6478c97f25cfb38c349ac [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:16:16.672325 14306 raft_consensus.cc:2454] T af429b0b011b406d98bf154b004313dc P ca07c3fe6ef6478c97f25cfb38c349ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f4ec3e52e1a494c82be7558bea4ff26 in term 1.
I20260924 04:16:16.672336 14440 raft_consensus.cc:2454] T af429b0b011b406d98bf154b004313dc P 683ba6100ecc4be5a17b03622dcad539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f4ec3e52e1a494c82be7558bea4ff26 in term 1.
I20260924 04:16:16.672564 14507 leader_election.cc:302] T af429b0b011b406d98bf154b004313dc P 3f4ec3e52e1a494c82be7558bea4ff26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f4ec3e52e1a494c82be7558bea4ff26, ca07c3fe6ef6478c97f25cfb38c349ac; no voters:
I20260924 04:16:16.672741 14644 raft_consensus.cc:2781] T af429b0b011b406d98bf154b004313dc P 3f4ec3e52e1a494c82be7558bea4ff26 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:16:16.672942 14644 raft_consensus.cc:686] T af429b0b011b406d98bf154b004313dc P 3f4ec3e52e1a494c82be7558bea4ff26 [term 1 LEADER]: Becoming Leader. State: Replica: 3f4ec3e52e1a494c82be7558bea4ff26, State: Running, Role: LEADER
I20260924 04:16:16.673243 14644 consensus_queue.cc:237] T af429b0b011b406d98bf154b004313dc P 3f4ec3e52e1a494c82be7558bea4ff26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "683ba6100ecc4be5a17b03622dcad539" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 } } peers { permanent_uuid: "3f4ec3e52e1a494c82be7558bea4ff26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36613 } } peers { permanent_uuid: "ca07c3fe6ef6478c97f25cfb38c349ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34429 } }
I20260924 04:16:16.676210 14160 catalog_manager.cc:5495] T af429b0b011b406d98bf154b004313dc P 3f4ec3e52e1a494c82be7558bea4ff26 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f4ec3e52e1a494c82be7558bea4ff26 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "3f4ec3e52e1a494c82be7558bea4ff26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "683ba6100ecc4be5a17b03622dcad539" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f4ec3e52e1a494c82be7558bea4ff26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca07c3fe6ef6478c97f25cfb38c349ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34429 } health_report { overall_health: UNKNOWN } } }
I20260924 04:16:16.696725 14635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:16.693455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35110 (local address 127.0.98.3:36613)
0924 04:16:16.693577 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:16.693581 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:16.693594 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:16:16.693684 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:16.693687 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:16.693737 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:16.693822 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:16:16.694063 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:16.694550 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:16.695395 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:16.695601 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:16.695684 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:16.695960 (+ 276us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:16:16.696374 (+ 414us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:16:16.696477 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:16.696546 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 04:16:16.696601 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 04:16:16.702234 14306 raft_consensus.cc:1261] T af429b0b011b406d98bf154b004313dc P ca07c3fe6ef6478c97f25cfb38c349ac [term 1 FOLLOWER]: Refusing update from remote peer 3f4ec3e52e1a494c82be7558bea4ff26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:16:16.702215 14440 raft_consensus.cc:1261] T af429b0b011b406d98bf154b004313dc P 683ba6100ecc4be5a17b03622dcad539 [term 1 FOLLOWER]: Refusing update from remote peer 3f4ec3e52e1a494c82be7558bea4ff26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:16:16.703040 14647 consensus_queue.cc:1035] T af429b0b011b406d98bf154b004313dc P 3f4ec3e52e1a494c82be7558bea4ff26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "683ba6100ecc4be5a17b03622dcad539" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:16:16.703271 14644 consensus_queue.cc:1035] T af429b0b011b406d98bf154b004313dc P 3f4ec3e52e1a494c82be7558bea4ff26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca07c3fe6ef6478c97f25cfb38c349ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:16:16.712909 14651 mvcc.cc:205] Tried to move back new op lower bound from 7332754950965579776 to 7332754950856257536. Current Snapshot: MvccSnapshot[applied={T|T < 7332754950965579776}]
I20260924 04:16:16.717923 14161 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:16:16.718071 14161 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:16:16.720248 14161 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a: Sending DeleteTablet for 3 replicas of tablet af429b0b011b406d98bf154b004313dc
I20260924 04:16:16.720983 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 14223
I20260924 04:16:16.720984 14554 tablet_service.cc:1514] Processing DeleteTablet for tablet af429b0b011b406d98bf154b004313dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:16:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35094
I20260924 04:16:16.721083 14286 tablet_service.cc:1514] Processing DeleteTablet for tablet af429b0b011b406d98bf154b004313dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:16:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39802
I20260924 04:16:16.722354 14659 tablet_replica.cc:330] T af429b0b011b406d98bf154b004313dc P 3f4ec3e52e1a494c82be7558bea4ff26: stopping tablet replica
I20260924 04:16:16.722345 14420 tablet_service.cc:1514] Processing DeleteTablet for tablet af429b0b011b406d98bf154b004313dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:16:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34424
I20260924 04:16:16.722783 14659 raft_consensus.cc:2229] T af429b0b011b406d98bf154b004313dc P 3f4ec3e52e1a494c82be7558bea4ff26 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:16:16.723143 14659 raft_consensus.cc:2258] T af429b0b011b406d98bf154b004313dc P 3f4ec3e52e1a494c82be7558bea4ff26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:16:16.724695 14659 ts_tablet_manager.cc:1905] T af429b0b011b406d98bf154b004313dc P 3f4ec3e52e1a494c82be7558bea4ff26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:16:16.728176 14659 ts_tablet_manager.cc:1918] T af429b0b011b406d98bf154b004313dc P 3f4ec3e52e1a494c82be7558bea4ff26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:16:16.728282 14659 log.cc:1196] T af429b0b011b406d98bf154b004313dc P 3f4ec3e52e1a494c82be7558bea4ff26: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-2/wal/wals/af429b0b011b406d98bf154b004313dc
I20260924 04:16:16.728572 14659 ts_tablet_manager.cc:1939] T af429b0b011b406d98bf154b004313dc P 3f4ec3e52e1a494c82be7558bea4ff26: Deleting consensus metadata
I20260924 04:16:16.729804 14145 catalog_manager.cc:4841] TS 3f4ec3e52e1a494c82be7558bea4ff26 (127.0.98.3:36613): tablet af429b0b011b406d98bf154b004313dc (table test-table [id=89fcc549a36945bf8a42ce746731cd7f]) successfully deleted
W20260924 04:16:16.730402 14145 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34429 (error 108)
I20260924 04:16:16.731009 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 14357
I20260924 04:16:16.731815 14632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:16.731149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34429 (local address 127.0.0.1:39804)
0924 04:16:16.731664 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:16.731731 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435}
W20260924 04:16:16.732933 14145 catalog_manager.cc:4584] TS ca07c3fe6ef6478c97f25cfb38c349ac (127.0.98.1:34429): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet af429b0b011b406d98bf154b004313dc: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34429: connect: Connection refused (error 111)
I20260924 04:16:16.737985 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 14491
I20260924 04:16:16.739384 14632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:16.738092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34529 (local address 127.0.0.1:34438)
0924 04:16:16.739159 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:16.739209 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:34529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":996}
I20260924 04:16:16.742573 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 14129
I20260924 04:16:16.747560 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43281
--webserver_interface=127.0.98.62
--webserver_port=34517
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43281
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:16:16.854558 14662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:16:16.854873 14662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:16:16.854946 14662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:16:16.858897 14662 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:16:16.858949 14662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:16:16.858978 14662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:16:16.858999 14662 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:16:16.859028 14662 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:16:16.864166 14662 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=34517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:16:16.865298 14662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:16:16.866257 14662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:16:16.878369 14669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:16:16.878360 14668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:16:16.878364 14671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:16:16.878595 14662 server_base.cc:1034] running on GCE node
I20260924 04:16:16.879223 14662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:16:16.880327 14662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:16:16.881480 14662 hybrid_clock.cc:648] HybridClock initialized: now 1790223376881463 us; error 33 us; skew 500 ppm
Sep 24 04:16:16 dist-test-slave-p401 krb5kdc[14116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223376, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:16:16.884081 14662 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:16:16.884944 14662 webserver.cc:460] Webserver started at http://127.0.98.62:34517/ using document root <none> and password file <none>
I20260924 04:16:16.885479 14662 fs_manager.cc:359] Metadata directory not provided
I20260924 04:16:16.885561 14662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:16:16.889386 14662 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260924 04:16:16.891474 14678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:16:16.892647 14662 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:16:16.892759 14662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/master-0/wal
uuid: "ff323d589be24ca68b0f38f5703db99a"
format_stamp: "Formatted at 2026-09-24 04:16:15 on dist-test-slave-p401"
server_key: "f46dbda456d0b077c86cd9443b374b7e"
server_key_iv: "424d079b8077f4ffaa5b90beb0460532"
server_key_version: "encryptionkey@0"
I20260924 04:16:16.893337 14662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:16:16.909950 14662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:16:16.913024 14662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:16:16.913194 14662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:16:16.919894 14662 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43281
I20260924 04:16:16.920151 14730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43281 every 8 connection(s)
I20260924 04:16:16.920943 14662 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:16:16.923082 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 14662
I20260924 04:16:16.923480 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:34429
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:43281
--webserver_port=44461
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:16:16.925205 14731 sys_catalog.cc:263] Verifying existing consensus state
I20260924 04:16:16.927311 14731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a: Bootstrap starting.
I20260924 04:16:16.949846 14731 log.cc:824] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a: Log is configured to *not* fsync() on all Append() calls
I20260924 04:16:16.960732 14731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:16:16.961180 14731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a: Bootstrap complete.
I20260924 04:16:16.966535 14731 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff323d589be24ca68b0f38f5703db99a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43281 } }
I20260924 04:16:16.967056 14731 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff323d589be24ca68b0f38f5703db99a, State: Initialized, Role: FOLLOWER
I20260924 04:16:16.967629 14731 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff323d589be24ca68b0f38f5703db99a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43281 } }
I20260924 04:16:16.967778 14731 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:16:16.967850 14731 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:16:16.967947 14731 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a [term 1 FOLLOWER]: Advancing to term 2
I20260924 04:16:16.970382 14731 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff323d589be24ca68b0f38f5703db99a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43281 } }
I20260924 04:16:16.970744 14731 leader_election.cc:302] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff323d589be24ca68b0f38f5703db99a; no voters:
I20260924 04:16:16.971113 14731 leader_election.cc:288] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 04:16:16.971280 14735 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a [term 2 FOLLOWER]: Leader election won for term 2
I20260924 04:16:16.971549 14735 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a [term 2 LEADER]: Becoming Leader. State: Replica: ff323d589be24ca68b0f38f5703db99a, State: Running, Role: LEADER
I20260924 04:16:16.971866 14735 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff323d589be24ca68b0f38f5703db99a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43281 } }
I20260924 04:16:16.972429 14731 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:16:16.972766 14737 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff323d589be24ca68b0f38f5703db99a. Latest consensus state: current_term: 2 leader_uuid: "ff323d589be24ca68b0f38f5703db99a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff323d589be24ca68b0f38f5703db99a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43281 } } }
I20260924 04:16:16.972903 14737 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a [sys.catalog]: This master's current role is: LEADER
I20260924 04:16:16.973198 14736 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ff323d589be24ca68b0f38f5703db99a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff323d589be24ca68b0f38f5703db99a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43281 } } }
I20260924 04:16:16.973286 14736 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a [sys.catalog]: This master's current role is: LEADER
I20260924 04:16:16.973578 14739 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:16:16.978247 14739 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:16:16.978856 14739 catalog_manager.cc:1259] Loaded cluster ID: ebd3bd98b35c4498a704f768bab163a6
I20260924 04:16:16.978922 14739 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:16:16.980042 14739 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260924 04:16:16.982751 14739 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260924 04:16:16.986917 14739 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:16:16.987617 14739 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a: Loaded TSK: 0
I20260924 04:16:16.988583 14739 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 04:16:17.059681 14733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:16:17.059940 14733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:16:17.059984 14733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:16:17.063918 14733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:16:17.063968 14733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:16:17.064029 14733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:16:17.069072 14733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:34429
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=44461
--tserver_master_addrs=127.0.98.62:43281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:16:17.070011 14733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:16:17.070955 14733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:16:17.080147 14758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:16:17.080164 14761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:16:17.080149 14759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:16:17.080466 14733 server_base.cc:1034] running on GCE node
I20260924 04:16:17.080818 14733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:16:17.081413 14733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:16:17.082566 14733 hybrid_clock.cc:648] HybridClock initialized: now 1790223377082540 us; error 44 us; skew 500 ppm
Sep 24 04:16:17 dist-test-slave-p401 krb5kdc[14116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:16:17.085230 14733 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:16:17.086082 14733 webserver.cc:460] Webserver started at http://127.0.98.1:44461/ using document root <none> and password file <none>
I20260924 04:16:17.086587 14733 fs_manager.cc:359] Metadata directory not provided
I20260924 04:16:17.086663 14733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:16:17.090281 14733 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:16:17.092285 14768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:16:17.093465 14733 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:16:17.093551 14733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-0/wal
uuid: "ca07c3fe6ef6478c97f25cfb38c349ac"
format_stamp: "Formatted at 2026-09-24 04:16:16 on dist-test-slave-p401"
server_key: "eb616c13f460039f1995842ba5d0fc85"
server_key_iv: "ba357bb10b7969a69c4c58212b96bd53"
server_key_version: "encryptionkey@0"
I20260924 04:16:17.094056 14733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:16:17.115401 14733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:16:17.118453 14733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:16:17.118625 14733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:16:17.119210 14733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:16:17.120285 14775 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 04:16:17.124181 14733 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 04:16:17.124245 14733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260924 04:16:17.124307 14733 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 04:16:17.125885 14733 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 04:16:17.125929 14733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260924 04:16:17.126178 14775 tablet_bootstrap.cc:492] T af429b0b011b406d98bf154b004313dc P ca07c3fe6ef6478c97f25cfb38c349ac: Bootstrap starting.
I20260924 04:16:17.134979 14733 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34429
I20260924 04:16:17.135042 14882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34429 every 8 connection(s)
I20260924 04:16:17.135885 14733 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:16:17.140317 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 14733
I20260924 04:16:17.140751 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34529
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:43281
--webserver_port=36609
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:16:17.144824 14775 log.cc:824] T af429b0b011b406d98bf154b004313dc P ca07c3fe6ef6478c97f25cfb38c349ac: Log is configured to *not* fsync() on all Append() calls
Sep 24 04:16:17 dist-test-slave-p401 krb5kdc[14116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:16:17.152315 14775 tablet_bootstrap.cc:492] T af429b0b011b406d98bf154b004313dc P ca07c3fe6ef6478c97f25cfb38c349ac: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:16:17.152696 14775 tablet_bootstrap.cc:492] T af429b0b011b406d98bf154b004313dc P ca07c3fe6ef6478c97f25cfb38c349ac: Bootstrap complete.
I20260924 04:16:17.153905 14775 ts_tablet_manager.cc:1397] T af429b0b011b406d98bf154b004313dc P ca07c3fe6ef6478c97f25cfb38c349ac: Time spent bootstrapping tablet: real 0.028s user 0.025s sys 0.000s
I20260924 04:16:17.154067 14886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:17.137909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43281 (local address 127.0.98.1:41823)
0924 04:16:17.138354 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:17.138405 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 04:16:17.139174 (+ 769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:16:17.140639 (+ 1465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:16:17.140647 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:16:17.141096 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:16:17.141717 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:17.141735 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:17.143710 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:17.143712 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:17.144557 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:17.144566 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:17.145385 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:17.145389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:17.145593 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:17.146238 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:16:17.146259 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:16:17.148099 (+ 1840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:16:17.150693 (+ 2594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:17.150698 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:17.150711 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:17.151057 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:17.151511 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:17.151514 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:17.151516 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:17.151688 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:17.152361 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:16:17.152367 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:16:17.152661 (+ 294us) client_negotiation.cc:769] Sending connection context
0924 04:16:17.152851 (+ 190us) client_negotiation.cc:240] Negotiation successful
0924 04:16:17.153161 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":113,"threads_started":1}
I20260924 04:16:17.154371 14885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:17.137944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41823 (local address 127.0.98.62:43281)
0924 04:16:17.138462 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:17.138486 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:17.138541 (+ 55us) server_negotiation.cc:407] Connection header received
0924 04:16:17.139623 (+ 1082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:17.139643 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:17.140145 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:17.140510 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:16:17.141912 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:17.143567 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:17.144674 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:17.145234 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:17.145399 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:17.148307 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:16:17.148314 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:16:17.148331 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:16:17.148374 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:16:17.150580 (+ 2206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:17.151174 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:17.151179 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:17.151185 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:17.151391 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:17.151783 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:17.151787 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:17.151789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:17.152221 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:16:17.152451 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:17.152983 (+ 532us) server_negotiation.cc:299] Negotiation successful
0924 04:16:17.153269 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":125,"threads_started":1}
I20260924 04:16:17.156435 14775 raft_consensus.cc:348] T af429b0b011b406d98bf154b004313dc P ca07c3fe6ef6478c97f25cfb38c349ac [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "683ba6100ecc4be5a17b03622dcad539" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 } } peers { permanent_uuid: "3f4ec3e52e1a494c82be7558bea4ff26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36613 } } peers { permanent_uuid: "ca07c3fe6ef6478c97f25cfb38c349ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34429 } }
I20260924 04:16:17.156850 14883 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43281
I20260924 04:16:17.156903 14775 raft_consensus.cc:729] T af429b0b011b406d98bf154b004313dc P ca07c3fe6ef6478c97f25cfb38c349ac [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca07c3fe6ef6478c97f25cfb38c349ac, State: Initialized, Role: FOLLOWER
I20260924 04:16:17.157054 14883 heartbeater.cc:464] Registering TS with master...
I20260924 04:16:17.157351 14775 consensus_queue.cc:260] T af429b0b011b406d98bf154b004313dc P ca07c3fe6ef6478c97f25cfb38c349ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "683ba6100ecc4be5a17b03622dcad539" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 } } peers { permanent_uuid: "3f4ec3e52e1a494c82be7558bea4ff26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36613 } } peers { permanent_uuid: "ca07c3fe6ef6478c97f25cfb38c349ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34429 } }
I20260924 04:16:17.157567 14883 heartbeater.cc:511] Master 127.0.98.62:43281 requested a full tablet report, sending...
I20260924 04:16:17.158294 14775 ts_tablet_manager.cc:1428] T af429b0b011b406d98bf154b004313dc P ca07c3fe6ef6478c97f25cfb38c349ac: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20260924 04:16:17.160132 14695 ts_manager.cc:194] Registered new tserver with Master: ca07c3fe6ef6478c97f25cfb38c349ac (127.0.98.1:34429)
I20260924 04:16:17.163977 14695 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41823
I20260924 04:16:17.167268 14883 heartbeater.cc:503] Master 127.0.98.62:43281 was elected leader, sending a full tablet report...
Sep 24 04:16:17 dist-test-slave-p401 krb5kdc[14116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223376, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:16:17.174885 14891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:17.163343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39816 (local address 127.0.98.1:34429)
0924 04:16:17.163990 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:17.164000 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:17.164022 (+ 22us) server_negotiation.cc:407] Connection header received
0924 04:16:17.164129 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:17.164134 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:17.164289 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:17.164435 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:16:17.164994 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:17.165688 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:17.166505 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:17.166840 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:17.166933 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:17.169373 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:16:17.169381 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:16:17.169391 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:16:17.169412 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:16:17.171692 (+ 2280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:17.172432 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:17.172438 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:17.172440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:17.172574 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:17.173004 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:17.173008 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:17.173009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:17.173385 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:16:17.173623 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:17.174186 (+ 563us) server_negotiation.cc:299] Negotiation successful
0924 04:16:17.174432 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":90,"threads_started":1}
I20260924 04:16:17.175207 14890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:17.163271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34429 (local address 127.0.0.1:39816)
0924 04:16:17.163779 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:17.163822 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 04:16:17.163969 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:16:17.164435 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:16:17.164440 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:16:17.164781 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:16:17.164886 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:17.164892 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:17.165793 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:17.165796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:17.166400 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:17.166409 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:17.166933 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:17.166951 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:17.167054 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:17.167583 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:16:17.167595 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:16:17.169242 (+ 1647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:16:17.171846 (+ 2604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:17.171854 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:17.171859 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:17.172249 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:17.172709 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:17.172712 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:17.172714 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:17.172855 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:17.173528 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:16:17.173534 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:16:17.173849 (+ 315us) client_negotiation.cc:769] Sending connection context
0924 04:16:17.174031 (+ 182us) client_negotiation.cc:240] Negotiation successful
0924 04:16:17.175045 (+ 1014us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":113,"threads_started":1}
I20260924 04:16:17.181272 14817 tablet_service.cc:1514] Processing DeleteTablet for tablet af429b0b011b406d98bf154b004313dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:16:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39816
I20260924 04:16:17.182358 14892 tablet_replica.cc:330] stopping tablet replica
I20260924 04:16:17.182686 14892 raft_consensus.cc:2229] T af429b0b011b406d98bf154b004313dc P ca07c3fe6ef6478c97f25cfb38c349ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:16:17.182976 14892 raft_consensus.cc:2258] T af429b0b011b406d98bf154b004313dc P ca07c3fe6ef6478c97f25cfb38c349ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:16:17.184134 14892 ts_tablet_manager.cc:1905] T af429b0b011b406d98bf154b004313dc P ca07c3fe6ef6478c97f25cfb38c349ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:16:17.188544 14892 ts_tablet_manager.cc:1918] T af429b0b011b406d98bf154b004313dc P ca07c3fe6ef6478c97f25cfb38c349ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:16:17.188653 14892 log.cc:1196] T af429b0b011b406d98bf154b004313dc P ca07c3fe6ef6478c97f25cfb38c349ac: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-0/wal/wals/af429b0b011b406d98bf154b004313dc
I20260924 04:16:17.188915 14892 ts_tablet_manager.cc:1939] T af429b0b011b406d98bf154b004313dc P ca07c3fe6ef6478c97f25cfb38c349ac: Deleting consensus metadata
I20260924 04:16:17.191044 14679 catalog_manager.cc:4841] TS ca07c3fe6ef6478c97f25cfb38c349ac (127.0.98.1:34429): tablet af429b0b011b406d98bf154b004313dc (table test-table [id=89fcc549a36945bf8a42ce746731cd7f]) successfully deleted
W20260924 04:16:17.260012 14887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:16:17.260258 14887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:16:17.260299 14887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:16:17.264320 14887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:16:17.264376 14887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:16:17.264433 14887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:16:17.269507 14887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34529
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36609
--tserver_master_addrs=127.0.98.62:43281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:16:17.270480 14887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:16:17.271492 14887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:16:17.280860 14898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:16:17.280966 14897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:16:17.280928 14887 server_base.cc:1034] running on GCE node
W20260924 04:16:17.280862 14900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:16:17.281440 14887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:16:17.282035 14887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:16:17.283313 14887 hybrid_clock.cc:648] HybridClock initialized: now 1790223377283288 us; error 31 us; skew 500 ppm
Sep 24 04:16:17 dist-test-slave-p401 krb5kdc[14116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:16:17.286052 14887 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:16:17.286942 14887 webserver.cc:460] Webserver started at http://127.0.98.2:36609/ using document root <none> and password file <none>
I20260924 04:16:17.287520 14887 fs_manager.cc:359] Metadata directory not provided
I20260924 04:16:17.287601 14887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:16:17.291211 14887 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:16:17.293237 14907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:16:17.294363 14887 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:16:17.294485 14887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-1/wal
uuid: "683ba6100ecc4be5a17b03622dcad539"
format_stamp: "Formatted at 2026-09-24 04:16:16 on dist-test-slave-p401"
server_key: "7620d90c84aef57877cdca256c5f9472"
server_key_iv: "4185d02c74de83aa921fd676ac1eb203"
server_key_version: "encryptionkey@0"
I20260924 04:16:17.294996 14887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:16:17.330231 14887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:16:17.333204 14887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:16:17.333397 14887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:16:17.333999 14887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:16:17.335040 14914 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 04:16:17.338865 14887 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 04:16:17.338930 14887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260924 04:16:17.339013 14887 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 04:16:17.340628 14887 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 04:16:17.340672 14887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260924 04:16:17.340920 14914 tablet_bootstrap.cc:492] T af429b0b011b406d98bf154b004313dc P 683ba6100ecc4be5a17b03622dcad539: Bootstrap starting.
I20260924 04:16:17.350306 14887 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34529
I20260924 04:16:17.350419 15021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34529 every 8 connection(s)
I20260924 04:16:17.351285 14887 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:16:17.353601 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 14887
I20260924 04:16:17.354032 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:36613
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:43281
--webserver_port=39025
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:16:17.357735 14914 log.cc:824] T af429b0b011b406d98bf154b004313dc P 683ba6100ecc4be5a17b03622dcad539: Log is configured to *not* fsync() on all Append() calls
Sep 24 04:16:17 dist-test-slave-p401 krb5kdc[14116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:16:17.365015 14914 tablet_bootstrap.cc:492] T af429b0b011b406d98bf154b004313dc P 683ba6100ecc4be5a17b03622dcad539: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:16:17.365384 14914 tablet_bootstrap.cc:492] T af429b0b011b406d98bf154b004313dc P 683ba6100ecc4be5a17b03622dcad539: Bootstrap complete.
I20260924 04:16:17.366423 14914 ts_tablet_manager.cc:1397] T af429b0b011b406d98bf154b004313dc P 683ba6100ecc4be5a17b03622dcad539: Time spent bootstrapping tablet: real 0.026s user 0.018s sys 0.004s
I20260924 04:16:17.366814 14885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:17.353125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47115 (local address 127.0.98.62:43281)
0924 04:16:17.353270 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:17.353273 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:17.354116 (+ 843us) server_negotiation.cc:407] Connection header received
0924 04:16:17.355419 (+ 1303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:17.355424 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:17.355485 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:17.355587 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:16:17.357064 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:17.357848 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:17.359469 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:17.359809 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:17.359924 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:17.362890 (+ 2966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:16:17.362895 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:16:17.362898 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:16:17.362933 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:16:17.364576 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:17.365183 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:17.365187 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:17.365188 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:17.365294 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:17.365663 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:17.365667 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:17.365669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:17.365903 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:16:17.366008 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:17.366523 (+ 515us) server_negotiation.cc:299] Negotiation successful
0924 04:16:17.366653 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 04:16:17.367702 15024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:17.353360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43281 (local address 127.0.98.2:47115)
0924 04:16:17.353950 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:17.354005 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 04:16:17.355136 (+ 1131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:16:17.355783 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:16:17.355792 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:16:17.356267 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:16:17.356910 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:17.356927 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:17.357992 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:17.357995 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:17.359351 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:17.359362 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:17.359954 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:17.359958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:17.360160 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:17.360852 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:16:17.360874 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:16:17.362683 (+ 1809us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:16:17.364696 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:17.364702 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:17.364713 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:17.365048 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:17.365397 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:17.365400 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:17.365402 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:17.365563 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:17.366004 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:16:17.366009 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:16:17.366285 (+ 276us) client_negotiation.cc:769] Sending connection context
0924 04:16:17.366476 (+ 191us) client_negotiation.cc:240] Negotiation successful
0924 04:16:17.366796 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":140,"threads_started":1}
I20260924 04:16:17.368845 15022 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43281
I20260924 04:16:17.369083 15022 heartbeater.cc:464] Registering TS with master...
I20260924 04:16:17.369122 14914 raft_consensus.cc:348] T af429b0b011b406d98bf154b004313dc P 683ba6100ecc4be5a17b03622dcad539 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "683ba6100ecc4be5a17b03622dcad539" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 } } peers { permanent_uuid: "3f4ec3e52e1a494c82be7558bea4ff26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36613 } } peers { permanent_uuid: "ca07c3fe6ef6478c97f25cfb38c349ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34429 } }
I20260924 04:16:17.369580 15022 heartbeater.cc:511] Master 127.0.98.62:43281 requested a full tablet report, sending...
I20260924 04:16:17.369587 14914 raft_consensus.cc:729] T af429b0b011b406d98bf154b004313dc P 683ba6100ecc4be5a17b03622dcad539 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 683ba6100ecc4be5a17b03622dcad539, State: Initialized, Role: FOLLOWER
I20260924 04:16:17.370072 14914 consensus_queue.cc:260] T af429b0b011b406d98bf154b004313dc P 683ba6100ecc4be5a17b03622dcad539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "683ba6100ecc4be5a17b03622dcad539" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 } } peers { permanent_uuid: "3f4ec3e52e1a494c82be7558bea4ff26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36613 } } peers { permanent_uuid: "ca07c3fe6ef6478c97f25cfb38c349ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34429 } }
I20260924 04:16:17.370929 14914 ts_tablet_manager.cc:1428] T af429b0b011b406d98bf154b004313dc P 683ba6100ecc4be5a17b03622dcad539: Time spent starting tablet: real 0.004s user 0.007s sys 0.000s
I20260924 04:16:17.371788 14695 ts_manager.cc:194] Registered new tserver with Master: 683ba6100ecc4be5a17b03622dcad539 (127.0.98.2:34529)
I20260924 04:16:17.372908 14695 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47115
I20260924 04:16:17.375126 15022 heartbeater.cc:503] Master 127.0.98.62:43281 was elected leader, sending a full tablet report...
Sep 24 04:16:17 dist-test-slave-p401 krb5kdc[14116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223376, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260924 04:16:17.382333 14890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:17.372655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34529 (local address 127.0.0.1:34452)
0924 04:16:17.372775 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:17.372789 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:16:17.372862 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:16:17.373800 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:16:17.373803 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:16:17.373951 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:16:17.374028 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:17.374033 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:17.375273 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:17.375277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:17.375875 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:17.375884 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:17.376439 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:17.376454 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:17.376562 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:17.377129 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:16:17.377143 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:16:17.378578 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:16:17.380773 (+ 2195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:17.380776 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:17.380778 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:17.380981 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:17.381284 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:17.381287 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:17.381288 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:17.381368 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:17.381855 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:16:17.381858 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:16:17.381955 (+ 97us) client_negotiation.cc:769] Sending connection context
0924 04:16:17.382046 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 04:16:17.382181 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 04:16:17.382467 15028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:17.373005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34452 (local address 127.0.98.2:34529)
0924 04:16:17.373402 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:17.373412 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:17.373431 (+ 19us) server_negotiation.cc:407] Connection header received
0924 04:16:17.373487 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:17.373492 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:17.373678 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:17.373795 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:16:17.374303 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:17.375128 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:17.375985 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:17.376351 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:17.376453 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:17.378745 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:16:17.378756 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:16:17.378772 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:16:17.378800 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:16:17.380615 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:17.381091 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:17.381097 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:17.381099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:17.381193 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:17.381444 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:17.381446 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:17.381448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:17.381733 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:16:17.381856 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:17.382138 (+ 282us) server_negotiation.cc:299] Negotiation successful
0924 04:16:17.382304 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":316,"thread_start_us":115,"threads_started":1}
I20260924 04:16:17.383639 14956 tablet_service.cc:1514] Processing DeleteTablet for tablet af429b0b011b406d98bf154b004313dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:16:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34452
I20260924 04:16:17.384645 15029 tablet_replica.cc:330] stopping tablet replica
I20260924 04:16:17.384961 15029 raft_consensus.cc:2229] T af429b0b011b406d98bf154b004313dc P 683ba6100ecc4be5a17b03622dcad539 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:16:17.385196 15029 raft_consensus.cc:2258] T af429b0b011b406d98bf154b004313dc P 683ba6100ecc4be5a17b03622dcad539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:16:17.386134 15029 ts_tablet_manager.cc:1905] T af429b0b011b406d98bf154b004313dc P 683ba6100ecc4be5a17b03622dcad539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:16:17.389559 15029 ts_tablet_manager.cc:1918] T af429b0b011b406d98bf154b004313dc P 683ba6100ecc4be5a17b03622dcad539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:16:17.389645 15029 log.cc:1196] T af429b0b011b406d98bf154b004313dc P 683ba6100ecc4be5a17b03622dcad539: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-1/wal/wals/af429b0b011b406d98bf154b004313dc
I20260924 04:16:17.389895 15029 ts_tablet_manager.cc:1939] T af429b0b011b406d98bf154b004313dc P 683ba6100ecc4be5a17b03622dcad539: Deleting consensus metadata
I20260924 04:16:17.391481 14680 catalog_manager.cc:4841] TS 683ba6100ecc4be5a17b03622dcad539 (127.0.98.2:34529): tablet af429b0b011b406d98bf154b004313dc (table test-table [id=89fcc549a36945bf8a42ce746731cd7f]) successfully deleted
W20260924 04:16:17.472003 15025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:16:17.472270 15025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:16:17.472327 15025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:16:17.476567 15025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:16:17.476630 15025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:16:17.476709 15025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:16:17.482201 15025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:36613
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39025
--tserver_master_addrs=127.0.98.62:43281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:16:17.483212 15025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:16:17.484242 15025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:16:17.493750 15037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:16:17.493805 15025 server_base.cc:1034] running on GCE node
W20260924 04:16:17.493821 15035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:16:17.493764 15034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:16:17.494344 15025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:16:17.494951 15025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:16:17.496105 15025 hybrid_clock.cc:648] HybridClock initialized: now 1790223377496063 us; error 55 us; skew 500 ppm
Sep 24 04:16:17 dist-test-slave-p401 krb5kdc[14116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:16:17.498719 15025 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:16:17.499577 15025 webserver.cc:460] Webserver started at http://127.0.98.3:39025/ using document root <none> and password file <none>
I20260924 04:16:17.500108 15025 fs_manager.cc:359] Metadata directory not provided
I20260924 04:16:17.500176 15025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:16:17.503789 15025 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:16:17.505767 15044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:16:17.506929 15025 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:16:17.507016 15025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-2/wal
uuid: "3f4ec3e52e1a494c82be7558bea4ff26"
format_stamp: "Formatted at 2026-09-24 04:16:16 on dist-test-slave-p401"
server_key: "53ede7cc0860d0b18f63e3bf39c5ab03"
server_key_iv: "afc2b0656bde7f3c45894ad4f3608f74"
server_key_version: "encryptionkey@0"
I20260924 04:16:17.507555 15025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:16:17.531960 15025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:16:17.535244 15025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:16:17.535437 15025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:16:17.536175 15025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:16:17.537110 15025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:16:17.537171 15025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:16:17.537222 15025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:16:17.537250 15025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:16:17.545678 15025 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36613
I20260924 04:16:17.545706 15157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36613 every 8 connection(s)
I20260924 04:16:17.546653 15025 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:16:17.552402 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 15025
Sep 24 04:16:17 dist-test-slave-p401 krb5kdc[14116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:16:17.560818 15161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:17.553020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33096 (local address 127.0.98.62:43281)
0924 04:16:17.553222 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:17.553226 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:17.553448 (+ 222us) server_negotiation.cc:407] Connection header received
0924 04:16:17.553570 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:17.553572 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:17.553606 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:17.553672 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:16:17.554511 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:17.555205 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:17.556271 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:17.556486 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:17.556562 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:17.557904 (+ 1342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:16:17.557908 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:16:17.557911 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:16:17.557948 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:16:17.559321 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:17.559701 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:17.559704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:17.559706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:17.559777 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:17.560016 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:17.560020 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:17.560021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:17.560178 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:16:17.560359 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:17.560510 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 04:16:17.560633 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":73,"threads_started":1}
I20260924 04:16:17.560873 14885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:17.548398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58879 (local address 127.0.98.62:43281)
0924 04:16:17.548539 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:17.548543 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:17.549438 (+ 895us) server_negotiation.cc:407] Connection header received
0924 04:16:17.550724 (+ 1286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:17.550726 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:17.550773 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:17.550872 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:16:17.552125 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:17.552856 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:17.554521 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:17.554860 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:17.554961 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:17.557551 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:16:17.557553 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:16:17.557555 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:16:17.557577 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:16:17.558910 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:17.559429 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:17.559433 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:17.559434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:17.559517 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:17.559826 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:17.559829 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:17.559831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:17.559993 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:16:17.560073 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:17.560546 (+ 473us) server_negotiation.cc:299] Negotiation successful
0924 04:16:17.560700 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":49}
I20260924 04:16:17.562047 15160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:17.548821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43281 (local address 127.0.98.3:58879)
0924 04:16:17.549287 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:17.549329 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 04:16:17.550539 (+ 1210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:16:17.551064 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:16:17.551071 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:16:17.551485 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:16:17.551988 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:17.552002 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:17.552994 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:17.552998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:17.554359 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:17.554369 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:17.554994 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:17.554996 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:17.555134 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:17.555697 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:16:17.555716 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:16:17.557393 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:16:17.559008 (+ 1615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:17.559013 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:17.559027 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:17.559306 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:17.559599 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:17.559602 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:17.559604 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:17.559735 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:17.560084 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:16:17.560090 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:16:17.560318 (+ 228us) client_negotiation.cc:769] Sending connection context
0924 04:16:17.560475 (+ 157us) client_negotiation.cc:240] Negotiation successful
0924 04:16:17.560783 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":110,"threads_started":1}
I20260924 04:16:17.563036 15158 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43281
I20260924 04:16:17.563280 15158 heartbeater.cc:464] Registering TS with master...
I20260924 04:16:17.563745 15158 heartbeater.cc:511] Master 127.0.98.62:43281 requested a full tablet report, sending...
I20260924 04:16:17.564631 14695 ts_manager.cc:194] Registered new tserver with Master: 3f4ec3e52e1a494c82be7558bea4ff26 (127.0.98.3:36613)
I20260924 04:16:17.565121 14695 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58879
I20260924 04:16:17.574031 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:16:17.577708 14885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:17.574572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33112 (local address 127.0.98.62:43281)
0924 04:16:17.574691 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:17.574696 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:17.574860 (+ 164us) server_negotiation.cc:407] Connection header received
0924 04:16:17.574960 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:17.574962 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:17.575052 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:17.575131 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:16:17.575343 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:17.575817 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:17.576598 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:17.576789 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:17.576882 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:17.577078 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:16:17.577437 (+ 359us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:16:17.577520 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:17.577546 (+ 26us) server_negotiation.cc:299] Negotiation successful
0924 04:16:17.577587 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 04:16:17.578298 14695 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:33112:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:16:17.581571 14695 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:16:17.588706 14956 tablet_service.cc:1467] Processing CreateTablet for tablet 2e3be8cfaa9b4c2c8aeb33b3fd49d74c (DEFAULT_TABLE table=test-table [id=f12b66a94a7f4b18baa0cd5029888a93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:16:17.588711 14817 tablet_service.cc:1467] Processing CreateTablet for tablet 2e3be8cfaa9b4c2c8aeb33b3fd49d74c (DEFAULT_TABLE table=test-table [id=f12b66a94a7f4b18baa0cd5029888a93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:16:17.589267 14956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e3be8cfaa9b4c2c8aeb33b3fd49d74c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:16:17.589291 14817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e3be8cfaa9b4c2c8aeb33b3fd49d74c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:16:17.591835 14775 tablet_bootstrap.cc:492] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac: Bootstrap starting.
I20260924 04:16:17.592258 14914 tablet_bootstrap.cc:492] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P 683ba6100ecc4be5a17b03622dcad539: Bootstrap starting.
I20260924 04:16:17.592676 14775 tablet_bootstrap.cc:654] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac: Neither blocks nor log segments found. Creating new log.
I20260924 04:16:17.593050 14914 tablet_bootstrap.cc:654] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P 683ba6100ecc4be5a17b03622dcad539: Neither blocks nor log segments found. Creating new log.
I20260924 04:16:17.593997 14890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:17.587935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36613 (local address 127.0.0.1:35120)
0924 04:16:17.588140 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:17.588154 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:16:17.588222 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:16:17.589039 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:16:17.589042 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:16:17.589060 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:16:17.589160 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:17.589167 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:17.591021 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:17.591024 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:17.592185 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:17.592195 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:17.593602 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:17.593606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:17.593697 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:17.593746 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 04:16:17.593823 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 04:16:17.593899 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136}
I20260924 04:16:17.594123 15167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:17.588158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35120 (local address 127.0.98.3:36613)
0924 04:16:17.588532 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:17.588543 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:17.588567 (+ 24us) server_negotiation.cc:407] Connection header received
0924 04:16:17.588643 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:17.588648 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:17.588868 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:17.589307 (+ 439us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:16:17.589502 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:17.590884 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:17.592399 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:17.593481 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:17.593588 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:17.593749 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:17.593883 (+ 134us) server_negotiation.cc:299] Negotiation successful
0924 04:16:17.593983 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":150,"threads_started":1}
I20260924 04:16:17.594574 14775 tablet_bootstrap.cc:492] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac: No bootstrap required, opened a new log
I20260924 04:16:17.594574 14914 tablet_bootstrap.cc:492] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P 683ba6100ecc4be5a17b03622dcad539: No bootstrap required, opened a new log
I20260924 04:16:17.594660 14914 ts_tablet_manager.cc:1397] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P 683ba6100ecc4be5a17b03622dcad539: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:16:17.594660 14775 ts_tablet_manager.cc:1397] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 04:16:17.595085 14775 raft_consensus.cc:348] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f4ec3e52e1a494c82be7558bea4ff26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36613 } } peers { permanent_uuid: "683ba6100ecc4be5a17b03622dcad539" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 } } peers { permanent_uuid: "ca07c3fe6ef6478c97f25cfb38c349ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34429 } }
I20260924 04:16:17.595085 14914 raft_consensus.cc:348] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P 683ba6100ecc4be5a17b03622dcad539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f4ec3e52e1a494c82be7558bea4ff26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36613 } } peers { permanent_uuid: "683ba6100ecc4be5a17b03622dcad539" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 } } peers { permanent_uuid: "ca07c3fe6ef6478c97f25cfb38c349ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34429 } }
I20260924 04:16:17.595211 14914 raft_consensus.cc:374] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P 683ba6100ecc4be5a17b03622dcad539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:16:17.595211 14775 raft_consensus.cc:374] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:16:17.595284 14775 raft_consensus.cc:729] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca07c3fe6ef6478c97f25cfb38c349ac, State: Initialized, Role: FOLLOWER
I20260924 04:16:17.595284 14914 raft_consensus.cc:729] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P 683ba6100ecc4be5a17b03622dcad539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 683ba6100ecc4be5a17b03622dcad539, State: Initialized, Role: FOLLOWER
I20260924 04:16:17.595407 14775 consensus_queue.cc:260] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f4ec3e52e1a494c82be7558bea4ff26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36613 } } peers { permanent_uuid: "683ba6100ecc4be5a17b03622dcad539" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 } } peers { permanent_uuid: "ca07c3fe6ef6478c97f25cfb38c349ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34429 } }
I20260924 04:16:17.595407 14914 consensus_queue.cc:260] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P 683ba6100ecc4be5a17b03622dcad539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f4ec3e52e1a494c82be7558bea4ff26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36613 } } peers { permanent_uuid: "683ba6100ecc4be5a17b03622dcad539" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 } } peers { permanent_uuid: "ca07c3fe6ef6478c97f25cfb38c349ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34429 } }
I20260924 04:16:17.595659 14914 ts_tablet_manager.cc:1428] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P 683ba6100ecc4be5a17b03622dcad539: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:16:17.596307 14775 ts_tablet_manager.cc:1428] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:16:17.597163 15092 tablet_service.cc:1467] Processing CreateTablet for tablet 2e3be8cfaa9b4c2c8aeb33b3fd49d74c (DEFAULT_TABLE table=test-table [id=f12b66a94a7f4b18baa0cd5029888a93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:16:17.598323 15092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e3be8cfaa9b4c2c8aeb33b3fd49d74c. 1 dirs total, 0 dirs full, 0 dirs failed
W20260924 04:16:17.602372 15023 tablet.cc:2369] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P 683ba6100ecc4be5a17b03622dcad539: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:16:17.605057 15170 tablet_bootstrap.cc:492] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P 3f4ec3e52e1a494c82be7558bea4ff26: Bootstrap starting.
I20260924 04:16:17.606897 15170 tablet_bootstrap.cc:654] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P 3f4ec3e52e1a494c82be7558bea4ff26: Neither blocks nor log segments found. Creating new log.
I20260924 04:16:17.608009 15170 log.cc:824] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P 3f4ec3e52e1a494c82be7558bea4ff26: Log is configured to *not* fsync() on all Append() calls
I20260924 04:16:17.609620 15170 tablet_bootstrap.cc:492] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P 3f4ec3e52e1a494c82be7558bea4ff26: No bootstrap required, opened a new log
I20260924 04:16:17.609769 15170 ts_tablet_manager.cc:1397] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P 3f4ec3e52e1a494c82be7558bea4ff26: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 04:16:17.612656 15170 raft_consensus.cc:348] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P 3f4ec3e52e1a494c82be7558bea4ff26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f4ec3e52e1a494c82be7558bea4ff26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36613 } } peers { permanent_uuid: "683ba6100ecc4be5a17b03622dcad539" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 } } peers { permanent_uuid: "ca07c3fe6ef6478c97f25cfb38c349ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34429 } }
I20260924 04:16:17.612922 15170 raft_consensus.cc:374] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P 3f4ec3e52e1a494c82be7558bea4ff26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:16:17.612982 15170 raft_consensus.cc:729] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P 3f4ec3e52e1a494c82be7558bea4ff26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f4ec3e52e1a494c82be7558bea4ff26, State: Initialized, Role: FOLLOWER
I20260924 04:16:17.613413 15170 consensus_queue.cc:260] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P 3f4ec3e52e1a494c82be7558bea4ff26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f4ec3e52e1a494c82be7558bea4ff26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36613 } } peers { permanent_uuid: "683ba6100ecc4be5a17b03622dcad539" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 } } peers { permanent_uuid: "ca07c3fe6ef6478c97f25cfb38c349ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34429 } }
I20260924 04:16:17.613970 15158 heartbeater.cc:503] Master 127.0.98.62:43281 was elected leader, sending a full tablet report...
I20260924 04:16:17.614192 15170 ts_tablet_manager.cc:1428] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P 3f4ec3e52e1a494c82be7558bea4ff26: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20260924 04:16:17.636938 14884 tablet.cc:2369] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:16:17.797559 15159 tablet.cc:2369] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P 3f4ec3e52e1a494c82be7558bea4ff26: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:16:17.873553 14889 raft_consensus.cc:482] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:16:17.873811 14889 raft_consensus.cc:504] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f4ec3e52e1a494c82be7558bea4ff26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36613 } } peers { permanent_uuid: "683ba6100ecc4be5a17b03622dcad539" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 } } peers { permanent_uuid: "ca07c3fe6ef6478c97f25cfb38c349ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34429 } }
I20260924 04:16:17.874950 14889 leader_election.cc:288] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f4ec3e52e1a494c82be7558bea4ff26 (127.0.98.3:36613), 683ba6100ecc4be5a17b03622dcad539 (127.0.98.2:34529)
I20260924 04:16:17.878849 15028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:17.875329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52545 (local address 127.0.98.2:34529)
0924 04:16:17.875552 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:17.875556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:17.875641 (+ 85us) server_negotiation.cc:407] Connection header received
0924 04:16:17.875825 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:17.875827 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:17.875880 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:17.875956 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:16:17.876221 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:17.876726 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:17.877742 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:17.878279 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:17.878373 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:17.878531 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:17.878657 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 04:16:17.878726 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260924 04:16:17.878872 15173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:17.875253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34529 (local address 127.0.98.1:52545)
0924 04:16:17.875554 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:17.875567 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:16:17.875695 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:16:17.875979 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:16:17.875982 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:16:17.875998 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:16:17.876119 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:17.876125 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:17.876829 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:17.876832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:17.877607 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:17.877615 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:17.878405 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:17.878409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:17.878508 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:17.878558 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 04:16:17.878632 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 04:16:17.878699 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":73,"threads_started":1}
I20260924 04:16:17.879165 15174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:17.875266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36613 (local address 127.0.98.1:51059)
0924 04:16:17.875673 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:17.875684 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:16:17.875994 (+ 310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:16:17.876248 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:16:17.876251 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:16:17.876266 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:16:17.876351 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:17.876355 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:17.877018 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:17.877021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:17.877901 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:17.877911 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:17.878795 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:17.878799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:17.878883 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:17.878905 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 04:16:17.878969 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 04:16:17.879033 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":71,"threads_started":1}
I20260924 04:16:17.879483 15167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:17.875400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51059 (local address 127.0.98.3:36613)
0924 04:16:17.875514 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:17.875519 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:17.875763 (+ 244us) server_negotiation.cc:407] Connection header received
0924 04:16:17.876124 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:17.876126 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:17.876171 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:17.876231 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:16:17.876448 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:17.876922 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:17.878026 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:17.878583 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:17.878677 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:17.878729 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:17.879299 (+ 570us) server_negotiation.cc:299] Negotiation successful
0924 04:16:17.879376 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 04:16:17.879561 14976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e3be8cfaa9b4c2c8aeb33b3fd49d74c" candidate_uuid: "ca07c3fe6ef6478c97f25cfb38c349ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "683ba6100ecc4be5a17b03622dcad539" is_pre_election: true
I20260924 04:16:17.879887 14976 raft_consensus.cc:2454] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P 683ba6100ecc4be5a17b03622dcad539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca07c3fe6ef6478c97f25cfb38c349ac in term 0.
I20260924 04:16:17.880111 15112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e3be8cfaa9b4c2c8aeb33b3fd49d74c" candidate_uuid: "ca07c3fe6ef6478c97f25cfb38c349ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f4ec3e52e1a494c82be7558bea4ff26" is_pre_election: true
I20260924 04:16:17.880324 14770 leader_election.cc:302] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 683ba6100ecc4be5a17b03622dcad539, ca07c3fe6ef6478c97f25cfb38c349ac; no voters:
I20260924 04:16:17.880385 15112 raft_consensus.cc:2454] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P 3f4ec3e52e1a494c82be7558bea4ff26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca07c3fe6ef6478c97f25cfb38c349ac in term 0.
I20260924 04:16:17.880577 14889 raft_consensus.cc:2781] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:16:17.880669 14889 raft_consensus.cc:482] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:16:17.880716 14889 raft_consensus.cc:3037] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:16:17.881670 14889 raft_consensus.cc:504] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f4ec3e52e1a494c82be7558bea4ff26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36613 } } peers { permanent_uuid: "683ba6100ecc4be5a17b03622dcad539" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 } } peers { permanent_uuid: "ca07c3fe6ef6478c97f25cfb38c349ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34429 } }
I20260924 04:16:17.881994 14889 leader_election.cc:288] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac [CANDIDATE]: Term 1 election: Requested vote from peers 3f4ec3e52e1a494c82be7558bea4ff26 (127.0.98.3:36613), 683ba6100ecc4be5a17b03622dcad539 (127.0.98.2:34529)
I20260924 04:16:17.882274 14976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e3be8cfaa9b4c2c8aeb33b3fd49d74c" candidate_uuid: "ca07c3fe6ef6478c97f25cfb38c349ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "683ba6100ecc4be5a17b03622dcad539"
I20260924 04:16:17.882273 15112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e3be8cfaa9b4c2c8aeb33b3fd49d74c" candidate_uuid: "ca07c3fe6ef6478c97f25cfb38c349ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f4ec3e52e1a494c82be7558bea4ff26"
I20260924 04:16:17.882383 14976 raft_consensus.cc:3037] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P 683ba6100ecc4be5a17b03622dcad539 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:16:17.882408 15112 raft_consensus.cc:3037] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P 3f4ec3e52e1a494c82be7558bea4ff26 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:16:17.883554 15112 raft_consensus.cc:2454] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P 3f4ec3e52e1a494c82be7558bea4ff26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca07c3fe6ef6478c97f25cfb38c349ac in term 1.
I20260924 04:16:17.883554 14976 raft_consensus.cc:2454] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P 683ba6100ecc4be5a17b03622dcad539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca07c3fe6ef6478c97f25cfb38c349ac in term 1.
I20260924 04:16:17.883790 14769 leader_election.cc:302] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f4ec3e52e1a494c82be7558bea4ff26, ca07c3fe6ef6478c97f25cfb38c349ac; no voters:
I20260924 04:16:17.883981 14889 raft_consensus.cc:2781] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:16:17.884137 14889 raft_consensus.cc:686] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac [term 1 LEADER]: Becoming Leader. State: Replica: ca07c3fe6ef6478c97f25cfb38c349ac, State: Running, Role: LEADER
I20260924 04:16:17.884372 14889 consensus_queue.cc:237] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f4ec3e52e1a494c82be7558bea4ff26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36613 } } peers { permanent_uuid: "683ba6100ecc4be5a17b03622dcad539" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 } } peers { permanent_uuid: "ca07c3fe6ef6478c97f25cfb38c349ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34429 } }
I20260924 04:16:17.887132 14694 catalog_manager.cc:5495] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac reported cstate change: term changed from 0 to 1, leader changed from <none> to ca07c3fe6ef6478c97f25cfb38c349ac (127.0.98.1). New cstate: current_term: 1 leader_uuid: "ca07c3fe6ef6478c97f25cfb38c349ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f4ec3e52e1a494c82be7558bea4ff26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "683ba6100ecc4be5a17b03622dcad539" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca07c3fe6ef6478c97f25cfb38c349ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34429 } health_report { overall_health: HEALTHY } } }
I20260924 04:16:17.958134 15178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:17.954602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39826 (local address 127.0.98.1:34429)
0924 04:16:17.954905 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:17.954911 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:17.954922 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:16:17.954993 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:17.954995 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:17.955046 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:17.955149 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:16:17.955421 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:17.955933 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:17.956757 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:17.956967 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:17.957050 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:17.957298 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:16:17.957727 (+ 429us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:16:17.957834 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:17.957921 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 04:16:17.957985 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":96,"threads_started":1}
I20260924 04:16:17.962016 14976 raft_consensus.cc:1261] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P 683ba6100ecc4be5a17b03622dcad539 [term 1 FOLLOWER]: Refusing update from remote peer ca07c3fe6ef6478c97f25cfb38c349ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:16:17.962019 15112 raft_consensus.cc:1261] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P 3f4ec3e52e1a494c82be7558bea4ff26 [term 1 FOLLOWER]: Refusing update from remote peer ca07c3fe6ef6478c97f25cfb38c349ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:16:17.962667 15175 consensus_queue.cc:1035] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "683ba6100ecc4be5a17b03622dcad539" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:16:17.962833 14889 consensus_queue.cc:1035] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f4ec3e52e1a494c82be7558bea4ff26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:16:17.967760 15179 mvcc.cc:205] Tried to move back new op lower bound from 7332754956126453760 to 7332754955816476672. Current Snapshot: MvccSnapshot[applied={T|T < 7332754956126453760}]
I20260924 04:16:17.969772 15180 mvcc.cc:205] Tried to move back new op lower bound from 7332754956126453760 to 7332754955816476672. Current Snapshot: MvccSnapshot[applied={T|T < 7332754956126453760}]
I20260924 04:16:17.977027 14694 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:33112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:16:17.977202 14694 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:33112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:16:17.979657 14694 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a: Sending DeleteTablet for 3 replicas of tablet 2e3be8cfaa9b4c2c8aeb33b3fd49d74c
I20260924 04:16:17.980302 14956 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e3be8cfaa9b4c2c8aeb33b3fd49d74c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:16:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34452
I20260924 04:16:17.980522 14817 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e3be8cfaa9b4c2c8aeb33b3fd49d74c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:16:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39816
I20260924 04:16:17.980773 15188 tablet_replica.cc:330] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P 683ba6100ecc4be5a17b03622dcad539: stopping tablet replica
I20260924 04:16:17.980719 15092 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e3be8cfaa9b4c2c8aeb33b3fd49d74c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:16:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35120
I20260924 04:16:17.980930 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 14733
I20260924 04:16:17.981048 15188 raft_consensus.cc:2229] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P 683ba6100ecc4be5a17b03622dcad539 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:16:17.981220 15188 raft_consensus.cc:2258] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P 683ba6100ecc4be5a17b03622dcad539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:16:17.981652 15190 tablet_replica.cc:330] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P 3f4ec3e52e1a494c82be7558bea4ff26: stopping tablet replica
I20260924 04:16:17.981978 15190 raft_consensus.cc:2229] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P 3f4ec3e52e1a494c82be7558bea4ff26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:16:17.982180 15188 ts_tablet_manager.cc:1905] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P 683ba6100ecc4be5a17b03622dcad539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:16:17.982306 15190 raft_consensus.cc:2258] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P 3f4ec3e52e1a494c82be7558bea4ff26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:16:17.984493 15190 ts_tablet_manager.cc:1905] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P 3f4ec3e52e1a494c82be7558bea4ff26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:16:17.984668 15188 ts_tablet_manager.cc:1918] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P 683ba6100ecc4be5a17b03622dcad539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:16:17.984742 15188 log.cc:1196] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P 683ba6100ecc4be5a17b03622dcad539: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-1/wal/wals/2e3be8cfaa9b4c2c8aeb33b3fd49d74c
I20260924 04:16:17.984985 15188 ts_tablet_manager.cc:1939] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P 683ba6100ecc4be5a17b03622dcad539: Deleting consensus metadata
I20260924 04:16:17.985656 14680 catalog_manager.cc:4841] TS 683ba6100ecc4be5a17b03622dcad539 (127.0.98.2:34529): tablet 2e3be8cfaa9b4c2c8aeb33b3fd49d74c (table test-table [id=f12b66a94a7f4b18baa0cd5029888a93]) successfully deleted
I20260924 04:16:17.986747 15190 ts_tablet_manager.cc:1918] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P 3f4ec3e52e1a494c82be7558bea4ff26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:16:17.986841 15190 log.cc:1196] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P 3f4ec3e52e1a494c82be7558bea4ff26: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-2/wal/wals/2e3be8cfaa9b4c2c8aeb33b3fd49d74c
I20260924 04:16:17.987161 15190 ts_tablet_manager.cc:1939] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P 3f4ec3e52e1a494c82be7558bea4ff26: Deleting consensus metadata
W20260924 04:16:17.987325 14679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34429 (error 108)
I20260924 04:16:17.987504 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 14887
I20260924 04:16:17.988318 14679 catalog_manager.cc:4841] TS 3f4ec3e52e1a494c82be7558bea4ff26 (127.0.98.3:36613): tablet 2e3be8cfaa9b4c2c8aeb33b3fd49d74c (table test-table [id=f12b66a94a7f4b18baa0cd5029888a93]) successfully deleted
I20260924 04:16:17.988862 14890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:17.988618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34429 (local address 127.0.0.1:39842)
0924 04:16:17.988734 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:17.988796 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 04:16:17.989034 14679 catalog_manager.cc:4584] TS ca07c3fe6ef6478c97f25cfb38c349ac (127.0.98.1:34429): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2e3be8cfaa9b4c2c8aeb33b3fd49d74c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34429: connect: Connection refused (error 111)
I20260924 04:16:17.992497 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 15025
I20260924 04:16:17.996994 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 14662
I20260924 04:16:18.002135 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43281
--webserver_interface=127.0.98.62
--webserver_port=34517
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43281
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:16:18.107957 15192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:16:18.108218 15192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:16:18.108273 15192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:16:18.112257 15192 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:16:18.112319 15192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:16:18.112344 15192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:16:18.112367 15192 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:16:18.112394 15192 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:16:18.117631 15192 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=34517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:16:18.118655 15192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:16:18.119658 15192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:16:18.127645 15197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:16:18.127712 15200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:16:18.127816 15198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:16:18.127719 15192 server_base.cc:1034] running on GCE node
I20260924 04:16:18.128525 15192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:16:18.129596 15192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:16:18.130744 15192 hybrid_clock.cc:648] HybridClock initialized: now 1790223378130718 us; error 38 us; skew 500 ppm
Sep 24 04:16:18 dist-test-slave-p401 krb5kdc[14116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:16:18.133275 15192 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:16:18.134061 15192 webserver.cc:460] Webserver started at http://127.0.98.62:34517/ using document root <none> and password file <none>
I20260924 04:16:18.134552 15192 fs_manager.cc:359] Metadata directory not provided
I20260924 04:16:18.134605 15192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:16:18.138281 15192 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:16:18.140458 15207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:16:18.141606 15192 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:16:18.141714 15192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/master-0/wal
uuid: "ff323d589be24ca68b0f38f5703db99a"
format_stamp: "Formatted at 2026-09-24 04:16:15 on dist-test-slave-p401"
server_key: "f46dbda456d0b077c86cd9443b374b7e"
server_key_iv: "424d079b8077f4ffaa5b90beb0460532"
server_key_version: "encryptionkey@0"
I20260924 04:16:18.142234 15192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:16:18.156049 15192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:16:18.158847 15192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:16:18.159003 15192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:16:18.165222 15192 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43281
I20260924 04:16:18.165470 15259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43281 every 8 connection(s)
I20260924 04:16:18.166198 15192 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:16:18.167495 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 15192
I20260924 04:16:18.167868 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:34429
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:43281
--webserver_port=44461
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:16:18.170675 15260 sys_catalog.cc:263] Verifying existing consensus state
I20260924 04:16:18.172297 15260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a: Bootstrap starting.
I20260924 04:16:18.188911 15260 log.cc:824] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a: Log is configured to *not* fsync() on all Append() calls
I20260924 04:16:18.199908 15260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:16:18.200248 15260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a: Bootstrap complete.
I20260924 04:16:18.204103 15260 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff323d589be24ca68b0f38f5703db99a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43281 } }
I20260924 04:16:18.204499 15260 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff323d589be24ca68b0f38f5703db99a, State: Initialized, Role: FOLLOWER
I20260924 04:16:18.204967 15260 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff323d589be24ca68b0f38f5703db99a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43281 } }
I20260924 04:16:18.205093 15260 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:16:18.205160 15260 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:16:18.205256 15260 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a [term 2 FOLLOWER]: Advancing to term 3
I20260924 04:16:18.207029 15260 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff323d589be24ca68b0f38f5703db99a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43281 } }
I20260924 04:16:18.207374 15260 leader_election.cc:302] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff323d589be24ca68b0f38f5703db99a; no voters:
I20260924 04:16:18.207671 15260 leader_election.cc:288] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 04:16:18.207813 15264 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a [term 3 FOLLOWER]: Leader election won for term 3
I20260924 04:16:18.208083 15264 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a [term 3 LEADER]: Becoming Leader. State: Replica: ff323d589be24ca68b0f38f5703db99a, State: Running, Role: LEADER
I20260924 04:16:18.208402 15264 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff323d589be24ca68b0f38f5703db99a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43281 } }
I20260924 04:16:18.208833 15260 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:16:18.209717 15265 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ff323d589be24ca68b0f38f5703db99a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff323d589be24ca68b0f38f5703db99a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43281 } } }
I20260924 04:16:18.209837 15265 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a [sys.catalog]: This master's current role is: LEADER
I20260924 04:16:18.210583 15266 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff323d589be24ca68b0f38f5703db99a. Latest consensus state: current_term: 3 leader_uuid: "ff323d589be24ca68b0f38f5703db99a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff323d589be24ca68b0f38f5703db99a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43281 } } }
I20260924 04:16:18.210686 15266 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a [sys.catalog]: This master's current role is: LEADER
I20260924 04:16:18.213292 15280 catalog_manager.cc:1259] Loaded cluster ID: ebd3bd98b35c4498a704f768bab163a6
I20260924 04:16:18.213342 15280 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a: loading cluster ID for follower catalog manager: success
W20260924 04:16:18.214131 15280 catalog_manager.cc:1576] T 00000000000000000000000000000000 P ff323d589be24ca68b0f38f5703db99a: acquiring CA information for follower catalog manager: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
W20260924 04:16:18.214188 15280 catalog_manager.cc:873] Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO: failed to prepare follower catalog manager, will retry
I20260924 04:16:18.214251 15273 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:16:18.216919 15273 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:16:18.217409 15273 catalog_manager.cc:1259] Loaded cluster ID: ebd3bd98b35c4498a704f768bab163a6
I20260924 04:16:18.217471 15273 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260924 04:16:18.218112 15273 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f3ee74439ad google::LogMessage::Fail() at ??:0
@ 0x7f3ee744688a google::LogMessage::SendToLog() at ??:0
@ 0x7f3ee74434de google::LogMessage::Flush() at ??:0
@ 0x7f3ee7444a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f3ef1d86fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f3ef1d87d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f3ef1d84579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f3ef1dc8640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f3ee96953d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f3ee9695c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f3ee96973f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f3ee9686512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f3eeacca6db start_thread at ??:0
@ 0x7f3ee5f4161f clone at ??:0
W20260924 04:16:18.299541 15262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:16:18.299782 15262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:16:18.299825 15262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:16:18.303761 15262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:16:18.303812 15262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:16:18.303881 15262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:16:18.309019 15262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:34429
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=44461
--tserver_master_addrs=127.0.98.62:43281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:16:18.310036 15262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:16:18.311105 15262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:16:18.320112 15304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:16:18.320297 15302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:16:18.320118 15301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:16:18.320410 15262 server_base.cc:1034] running on GCE node
I20260924 04:16:18.320756 15262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:16:18.321357 15262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:16:18.322525 15262 hybrid_clock.cc:648] HybridClock initialized: now 1790223378322505 us; error 32 us; skew 500 ppm
Sep 24 04:16:18 dist-test-slave-p401 krb5kdc[14116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:16:18.325357 15262 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:16:18.326416 15262 webserver.cc:460] Webserver started at http://127.0.98.1:44461/ using document root <none> and password file <none>
I20260924 04:16:18.327124 15262 fs_manager.cc:359] Metadata directory not provided
I20260924 04:16:18.327219 15262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:16:18.332223 15262 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20260924 04:16:18.334861 15311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:16:18.336362 15262 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 04:16:18.336484 15262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-0/wal
uuid: "ca07c3fe6ef6478c97f25cfb38c349ac"
format_stamp: "Formatted at 2026-09-24 04:16:16 on dist-test-slave-p401"
server_key: "eb616c13f460039f1995842ba5d0fc85"
server_key_iv: "ba357bb10b7969a69c4c58212b96bd53"
server_key_version: "encryptionkey@0"
I20260924 04:16:18.337195 15262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:16:18.357884 15262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:16:18.361197 15262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:16:18.361394 15262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:16:18.361994 15262 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:16:18.363123 15318 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 04:16:18.368492 15262 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 04:16:18.368571 15262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260924 04:16:18.368639 15262 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 04:16:18.370268 15262 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 04:16:18.370317 15262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260924 04:16:18.370625 15318 tablet_bootstrap.cc:492] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac: Bootstrap starting.
I20260924 04:16:18.380808 15262 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34429
I20260924 04:16:18.380873 15425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34429 every 8 connection(s)
I20260924 04:16:18.381771 15262 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:16:18.384850 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 15262
I20260924 04:16:18.385367 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34529
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:43281
--webserver_port=36609
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:16:18.393422 15318 log.cc:824] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac: Log is configured to *not* fsync() on all Append() calls
I20260924 04:16:18.402765 15318 tablet_bootstrap.cc:492] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:16:18.403177 15318 tablet_bootstrap.cc:492] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac: Bootstrap complete.
I20260924 04:16:18.404579 15318 ts_tablet_manager.cc:1397] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac: Time spent bootstrapping tablet: real 0.034s user 0.024s sys 0.007s
I20260924 04:16:18.407995 15318 raft_consensus.cc:348] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f4ec3e52e1a494c82be7558bea4ff26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36613 } } peers { permanent_uuid: "683ba6100ecc4be5a17b03622dcad539" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 } } peers { permanent_uuid: "ca07c3fe6ef6478c97f25cfb38c349ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34429 } }
I20260924 04:16:18.408524 15318 raft_consensus.cc:729] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca07c3fe6ef6478c97f25cfb38c349ac, State: Initialized, Role: FOLLOWER
I20260924 04:16:18.409049 15318 consensus_queue.cc:260] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f4ec3e52e1a494c82be7558bea4ff26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36613 } } peers { permanent_uuid: "683ba6100ecc4be5a17b03622dcad539" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 } } peers { permanent_uuid: "ca07c3fe6ef6478c97f25cfb38c349ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34429 } }
I20260924 04:16:18.410115 15318 ts_tablet_manager.cc:1428] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260924 04:16:18.456169 15428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:16:18.383693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43281 (local address 127.0.98.1:43495)
0924 04:16:18.384185 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:18.384239 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 04:16:18.385324 (+ 1085us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:16:18.455445 (+ 70121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43281: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":125,"threads_started":1}
W20260924 04:16:18.456718 15426 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43281 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:43281: Client connection negotiation failed: client connection to 127.0.98.62:43281: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 04:16:18.457474 15428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:18.457219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43281 (local address 127.0.98.1:51953)
0924 04:16:18.457355 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:18.457418 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 04:16:18.458082 15428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:18.457897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43281 (local address 127.0.98.1:56525)
0924 04:16:18.457993 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:18.458034 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 04:16:18.458457 15426 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 04:16:18.502791 15429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:16:18.503077 15429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:16:18.503137 15429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:16:18.507256 15429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:16:18.507314 15429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:16:18.507395 15429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:16:18.512773 15429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34529
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36609
--tserver_master_addrs=127.0.98.62:43281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:16:18.513788 15429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:16:18.514810 15429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:16:18.523784 15436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:16:18.523823 15429 server_base.cc:1034] running on GCE node
W20260924 04:16:18.523851 15437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:16:18.523784 15439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:16:18.524390 15429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:16:18.525008 15429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:16:18.526167 15429 hybrid_clock.cc:648] HybridClock initialized: now 1790223378526143 us; error 35 us; skew 500 ppm
Sep 24 04:16:18 dist-test-slave-p401 krb5kdc[14116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:16:18.528738 15429 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:16:18.529626 15429 webserver.cc:460] Webserver started at http://127.0.98.2:36609/ using document root <none> and password file <none>
I20260924 04:16:18.530122 15429 fs_manager.cc:359] Metadata directory not provided
I20260924 04:16:18.530177 15429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:16:18.533833 15429 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:16:18.535821 15446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:16:18.536972 15429 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:16:18.537066 15429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-1/wal
uuid: "683ba6100ecc4be5a17b03622dcad539"
format_stamp: "Formatted at 2026-09-24 04:16:16 on dist-test-slave-p401"
server_key: "7620d90c84aef57877cdca256c5f9472"
server_key_iv: "4185d02c74de83aa921fd676ac1eb203"
server_key_version: "encryptionkey@0"
I20260924 04:16:18.537595 15429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:16:18.577172 15429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:16:18.579949 15429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:16:18.580114 15429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:16:18.580691 15429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:16:18.581496 15429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:16:18.581542 15429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:16:18.581584 15429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:16:18.581612 15429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:16:18.589974 15429 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34529
I20260924 04:16:18.590000 15559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34529 every 8 connection(s)
I20260924 04:16:18.590934 15429 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:16:18.594507 15562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:18.592914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43281 (local address 127.0.98.2:36355)
0924 04:16:18.593336 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:18.593813 (+ 477us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":102,"threads_started":1}
W20260924 04:16:18.594898 15560 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43281 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:43281: Client connection negotiation failed: client connection to 127.0.98.62:43281: connect: Connection refused (error 111)
I20260924 04:16:18.595419 15562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:18.595201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43281 (local address 127.0.98.2:33557)
0924 04:16:18.595317 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:18.595367 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 04:16:18.595899 15562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:18.595733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43281 (local address 127.0.98.2:33373)
0924 04:16:18.595815 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:18.595853 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 04:16:18.596001 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 15429
W20260924 04:16:18.596043 15560 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 04:16:18.596365 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:36613
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:43281
--webserver_port=39025
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:16:18.699844 15563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:16:18.700093 15563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:16:18.700134 15563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:16:18.704107 15563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:16:18.704166 15563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:16:18.704226 15563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:16:18.709336 15563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:36613
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39025
--tserver_master_addrs=127.0.98.62:43281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:16:18.710306 15563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:16:18.711300 15563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:16:18.720520 15568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:16:18.720530 15571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:16:18.720619 15563 server_base.cc:1034] running on GCE node
W20260924 04:16:18.720535 15569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:16:18.721378 15563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:16:18.722105 15563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:16:18.723315 15563 hybrid_clock.cc:648] HybridClock initialized: now 1790223378723296 us; error 34 us; skew 500 ppm
Sep 24 04:16:18 dist-test-slave-p401 krb5kdc[14116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:16:18.726693 15563 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:16:18.727715 15563 webserver.cc:460] Webserver started at http://127.0.98.3:39025/ using document root <none> and password file <none>
I20260924 04:16:18.728320 15563 fs_manager.cc:359] Metadata directory not provided
I20260924 04:16:18.728418 15563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:16:18.732573 15563 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260924 04:16:18.734454 15578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:16:18.735644 15563 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:16:18.735734 15563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-2/wal
uuid: "3f4ec3e52e1a494c82be7558bea4ff26"
format_stamp: "Formatted at 2026-09-24 04:16:16 on dist-test-slave-p401"
server_key: "53ede7cc0860d0b18f63e3bf39c5ab03"
server_key_iv: "afc2b0656bde7f3c45894ad4f3608f74"
server_key_version: "encryptionkey@0"
I20260924 04:16:18.736270 15563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:16:18.759766 15563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:16:18.762509 15563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:16:18.762670 15563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:16:18.763262 15563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:16:18.764074 15563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:16:18.764120 15563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:16:18.764160 15563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:16:18.764189 15563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:16:18.772336 15563 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36613
I20260924 04:16:18.772361 15691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36613 every 8 connection(s)
I20260924 04:16:18.773279 15563 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:16:18.776926 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:18.775312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43281 (local address 127.0.98.3:34771)
0924 04:16:18.775747 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:18.776216 (+ 469us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":103,"threads_started":1}
W20260924 04:16:18.777324 15692 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43281 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:43281: Client connection negotiation failed: client connection to 127.0.98.62:43281: connect: Connection refused (error 111)
I20260924 04:16:18.777849 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:18.777658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43281 (local address 127.0.98.3:35751)
0924 04:16:18.777751 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:18.777796 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 04:16:18.778402 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:18.778231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43281 (local address 127.0.98.3:34445)
0924 04:16:18.778320 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:18.778357 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 04:16:18.778539 15692 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 04:16:18.781936 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 15563
I20260924 04:16:19.459429 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:19.458988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43281 (local address 127.0.98.1:35679)
0924 04:16:19.459224 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:19.459310 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":69,"threads_started":1}
I20260924 04:16:19.596903 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:19.596534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43281 (local address 127.0.98.2:37169)
0924 04:16:19.596750 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:19.596804 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":67,"threads_started":1}
I20260924 04:16:19.671700 15698 raft_consensus.cc:482] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:16:19.671916 15698 raft_consensus.cc:504] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f4ec3e52e1a494c82be7558bea4ff26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36613 } } peers { permanent_uuid: "683ba6100ecc4be5a17b03622dcad539" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 } } peers { permanent_uuid: "ca07c3fe6ef6478c97f25cfb38c349ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34429 } }
I20260924 04:16:19.672907 15698 leader_election.cc:288] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f4ec3e52e1a494c82be7558bea4ff26 (127.0.98.3:36613), 683ba6100ecc4be5a17b03622dcad539 (127.0.98.2:34529)
Sep 24 04:16:19 dist-test-slave-p401 krb5kdc[14116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 24 04:16:19 dist-test-slave-p401 krb5kdc[14116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260924 04:16:19.686120 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:19.673131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34529 (local address 127.0.98.1:33637)
0924 04:16:19.673319 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:19.673334 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:16:19.673442 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:16:19.675110 (+ 1668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:16:19.675116 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:16:19.675687 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:16:19.676179 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:19.676193 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:19.677406 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:19.677408 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:19.678277 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:19.678289 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:19.679143 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:19.679147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:19.679345 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:19.679976 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:16:19.679987 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:16:19.681862 (+ 1875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:16:19.683855 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:19.683857 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:19.683859 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:19.684093 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:19.684395 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:19.684398 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:19.684400 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:19.684545 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:19.685273 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:16:19.685275 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:16:19.685420 (+ 145us) client_negotiation.cc:769] Sending connection context
0924 04:16:19.685581 (+ 161us) client_negotiation.cc:240] Negotiation successful
0924 04:16:19.685928 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":143,"spinlock_wait_cycles":31872}
I20260924 04:16:19.686143 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:19.673696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51381 (local address 127.0.98.3:36613)
0924 04:16:19.674033 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:19.674042 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:19.674072 (+ 30us) server_negotiation.cc:407] Connection header received
0924 04:16:19.674667 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:19.674672 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:19.674979 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:19.675216 (+ 237us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:16:19.676335 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:19.677301 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:19.678376 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:19.678972 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:19.679121 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:19.681852 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:16:19.681859 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:16:19.681875 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:16:19.681907 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:16:19.683600 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:19.684170 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:19.684176 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:19.684182 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:19.684297 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:19.684634 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:19.684637 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:19.684639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:19.685046 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:16:19.685145 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:19.685710 (+ 565us) server_negotiation.cc:299] Negotiation successful
0924 04:16:19.685990 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":79,"threads_started":1}
I20260924 04:16:19.686173 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:19.673281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33637 (local address 127.0.98.2:34529)
0924 04:16:19.673686 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:19.673707 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:19.673744 (+ 37us) server_negotiation.cc:407] Connection header received
0924 04:16:19.674413 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:19.674421 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:19.674751 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:19.675033 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:16:19.676341 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:19.677278 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:19.678413 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:19.679030 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:19.679155 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:19.682021 (+ 2866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:16:19.682025 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:16:19.682035 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:16:19.682061 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:16:19.683741 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:19.684193 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:19.684198 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:19.684202 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:19.684307 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:19.684631 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:19.684634 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:19.684636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:19.685047 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:16:19.685303 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:19.685715 (+ 412us) server_negotiation.cc:299] Negotiation successful
0924 04:16:19.685990 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":68,"threads_started":1}
I20260924 04:16:19.686162 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:19.673131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36613 (local address 127.0.98.1:51381)
0924 04:16:19.673433 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:19.673445 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:16:19.673541 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:16:19.675180 (+ 1639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:16:19.675183 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:16:19.675693 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:16:19.676179 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:19.676193 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:19.677416 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:19.677419 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:19.678268 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:19.678277 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:19.679322 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:19.679325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:19.679392 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:19.679921 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:16:19.679937 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:16:19.681585 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:16:19.683710 (+ 2125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:19.683717 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:19.683731 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:19.684069 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:19.684396 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:19.684399 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:19.684401 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:19.684546 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:19.685168 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:16:19.685175 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:16:19.685423 (+ 248us) client_negotiation.cc:769] Sending connection context
0924 04:16:19.685582 (+ 159us) client_negotiation.cc:240] Negotiation successful
0924 04:16:19.686001 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":250,"spinlock_wait_cycles":35968,"thread_start_us":72,"threads_started":1}
W20260924 04:16:19.689143 15312 leader_election.cc:341] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f4ec3e52e1a494c82be7558bea4ff26 (127.0.98.3:36613): Not found: Tablet not found: 2e3be8cfaa9b4c2c8aeb33b3fd49d74c
W20260924 04:16:19.689692 15313 leader_election.cc:341] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 683ba6100ecc4be5a17b03622dcad539 (127.0.98.2:34529): Not found: Tablet not found: 2e3be8cfaa9b4c2c8aeb33b3fd49d74c
I20260924 04:16:19.689790 15313 leader_election.cc:302] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca07c3fe6ef6478c97f25cfb38c349ac; no voters: 3f4ec3e52e1a494c82be7558bea4ff26, 683ba6100ecc4be5a17b03622dcad539
I20260924 04:16:19.690042 15698 raft_consensus.cc:2726] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 04:16:19.779501 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:19.779076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43281 (local address 127.0.98.3:40015)
0924 04:16:19.779352 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:19.779409 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":80,"threads_started":1}
I20260924 04:16:20.460547 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:20.460089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43281 (local address 127.0.98.1:37015)
0924 04:16:20.460386 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:20.460449 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":73,"threads_started":1}
I20260924 04:16:20.598091 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:20.597640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43281 (local address 127.0.98.2:47519)
0924 04:16:20.597933 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:20.597997 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":76,"threads_started":1}
I20260924 04:16:20.780669 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:20.780266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43281 (local address 127.0.98.3:54067)
0924 04:16:20.780511 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:20.780567 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":79,"threads_started":1}
I20260924 04:16:21.461591 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:21.461212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43281 (local address 127.0.98.1:47303)
0924 04:16:21.461445 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:21.461498 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":75,"threads_started":1}
I20260924 04:16:21.599277 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:21.598823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43281 (local address 127.0.98.2:56519)
0924 04:16:21.599083 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:21.599147 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":83,"threads_started":1}
I20260924 04:16:21.644618 15708 raft_consensus.cc:482] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:16:21.644739 15708 raft_consensus.cc:504] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f4ec3e52e1a494c82be7558bea4ff26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36613 } } peers { permanent_uuid: "683ba6100ecc4be5a17b03622dcad539" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 } } peers { permanent_uuid: "ca07c3fe6ef6478c97f25cfb38c349ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34429 } }
I20260924 04:16:21.645160 15708 leader_election.cc:288] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f4ec3e52e1a494c82be7558bea4ff26 (127.0.98.3:36613), 683ba6100ecc4be5a17b03622dcad539 (127.0.98.2:34529)
W20260924 04:16:21.645880 15313 leader_election.cc:341] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 683ba6100ecc4be5a17b03622dcad539 (127.0.98.2:34529): Not found: Tablet not found: 2e3be8cfaa9b4c2c8aeb33b3fd49d74c
W20260924 04:16:21.645974 15312 leader_election.cc:341] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f4ec3e52e1a494c82be7558bea4ff26 (127.0.98.3:36613): Not found: Tablet not found: 2e3be8cfaa9b4c2c8aeb33b3fd49d74c
I20260924 04:16:21.646060 15312 leader_election.cc:302] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca07c3fe6ef6478c97f25cfb38c349ac; no voters: 3f4ec3e52e1a494c82be7558bea4ff26, 683ba6100ecc4be5a17b03622dcad539
I20260924 04:16:21.646188 15708 raft_consensus.cc:2726] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 04:16:21.782016 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:21.781467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43281 (local address 127.0.98.3:35765)
0924 04:16:21.781825 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:21.781910 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":91,"threads_started":1}
I20260924 04:16:22.462807 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:22.462363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43281 (local address 127.0.98.1:47307)
0924 04:16:22.462638 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:22.462702 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":74,"threads_started":1}
I20260924 04:16:22.600445 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:22.600011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43281 (local address 127.0.98.2:35981)
0924 04:16:22.600280 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:22.600345 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":71,"threads_started":1}
I20260924 04:16:22.783329 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:22.782845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43281 (local address 127.0.98.3:38223)
0924 04:16:22.783120 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:22.783182 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":73,"threads_started":1}
I20260924 04:16:23.464085 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:23.463651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43281 (local address 127.0.98.1:49563)
0924 04:16:23.463934 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:23.463990 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":71,"threads_started":1}
I20260924 04:16:23.601608 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:23.601187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43281 (local address 127.0.98.2:45803)
0924 04:16:23.601448 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:23.601503 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":85,"threads_started":1}
I20260924 04:16:23.784463 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:23.784059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43281 (local address 127.0.98.3:33909)
0924 04:16:23.784314 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:23.784371 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":74,"threads_started":1}
I20260924 04:16:24.465271 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:24.464841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43281 (local address 127.0.98.1:41725)
0924 04:16:24.465100 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:24.465162 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":73,"threads_started":1}
I20260924 04:16:24.602914 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:24.602418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43281 (local address 127.0.98.2:38097)
0924 04:16:24.602719 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:24.602817 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":98,"threads_started":1}
I20260924 04:16:24.782945 15718 raft_consensus.cc:482] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:16:24.783068 15718 raft_consensus.cc:504] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f4ec3e52e1a494c82be7558bea4ff26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36613 } } peers { permanent_uuid: "683ba6100ecc4be5a17b03622dcad539" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 } } peers { permanent_uuid: "ca07c3fe6ef6478c97f25cfb38c349ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34429 } }
I20260924 04:16:24.783500 15718 leader_election.cc:288] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f4ec3e52e1a494c82be7558bea4ff26 (127.0.98.3:36613), 683ba6100ecc4be5a17b03622dcad539 (127.0.98.2:34529)
W20260924 04:16:24.784267 15312 leader_election.cc:341] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f4ec3e52e1a494c82be7558bea4ff26 (127.0.98.3:36613): Not found: Tablet not found: 2e3be8cfaa9b4c2c8aeb33b3fd49d74c
W20260924 04:16:24.784392 15313 leader_election.cc:341] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 683ba6100ecc4be5a17b03622dcad539 (127.0.98.2:34529): Not found: Tablet not found: 2e3be8cfaa9b4c2c8aeb33b3fd49d74c
I20260924 04:16:24.784458 15313 leader_election.cc:302] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca07c3fe6ef6478c97f25cfb38c349ac; no voters: 3f4ec3e52e1a494c82be7558bea4ff26, 683ba6100ecc4be5a17b03622dcad539
I20260924 04:16:24.784592 15718 raft_consensus.cc:2726] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 04:16:24.785795 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:24.785191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43281 (local address 127.0.98.3:43677)
0924 04:16:24.785625 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:24.785687 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":249,"threads_started":1}
I20260924 04:16:25.466475 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:25.466026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43281 (local address 127.0.98.1:53091)
0924 04:16:25.466321 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:25.466381 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":72,"threads_started":1}
I20260924 04:16:25.604084 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:25.603669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43281 (local address 127.0.98.2:42027)
0924 04:16:25.603935 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:25.603991 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":72,"threads_started":1}
I20260924 04:16:25.786973 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:25.786535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43281 (local address 127.0.98.3:59037)
0924 04:16:25.786821 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:25.786878 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":69,"threads_started":1}
I20260924 04:16:26.467849 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:26.467311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43281 (local address 127.0.98.1:35603)
0924 04:16:26.467618 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:26.467675 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":105,"threads_started":1}
I20260924 04:16:26.605367 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:26.604933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43281 (local address 127.0.98.2:48651)
0924 04:16:26.605204 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:26.605271 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":78,"threads_started":1}
I20260924 04:16:26.674444 15725 raft_consensus.cc:482] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:16:26.674575 15725 raft_consensus.cc:504] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f4ec3e52e1a494c82be7558bea4ff26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36613 } } peers { permanent_uuid: "683ba6100ecc4be5a17b03622dcad539" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 } } peers { permanent_uuid: "ca07c3fe6ef6478c97f25cfb38c349ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34429 } }
I20260924 04:16:26.674957 15725 leader_election.cc:288] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f4ec3e52e1a494c82be7558bea4ff26 (127.0.98.3:36613), 683ba6100ecc4be5a17b03622dcad539 (127.0.98.2:34529)
W20260924 04:16:26.675714 15312 leader_election.cc:341] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f4ec3e52e1a494c82be7558bea4ff26 (127.0.98.3:36613): Not found: Tablet not found: 2e3be8cfaa9b4c2c8aeb33b3fd49d74c
W20260924 04:16:26.675830 15313 leader_election.cc:341] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 683ba6100ecc4be5a17b03622dcad539 (127.0.98.2:34529): Not found: Tablet not found: 2e3be8cfaa9b4c2c8aeb33b3fd49d74c
I20260924 04:16:26.675913 15313 leader_election.cc:302] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca07c3fe6ef6478c97f25cfb38c349ac; no voters: 3f4ec3e52e1a494c82be7558bea4ff26, 683ba6100ecc4be5a17b03622dcad539
I20260924 04:16:26.676039 15725 raft_consensus.cc:2726] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 04:16:26.788158 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:26.787707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43281 (local address 127.0.98.3:49647)
0924 04:16:26.787979 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:26.788051 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":71,"threads_started":1}
I20260924 04:16:27.469049 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:27.468620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43281 (local address 127.0.98.1:38141)
0924 04:16:27.468893 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:27.468956 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":75,"threads_started":1}
I20260924 04:16:27.606632 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:27.606175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43281 (local address 127.0.98.2:38511)
0924 04:16:27.606460 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:27.606523 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":85,"threads_started":1}
I20260924 04:16:27.789386 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:27.788937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43281 (local address 127.0.98.3:34881)
0924 04:16:27.789219 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:27.789283 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":92,"threads_started":1}
I20260924 04:16:28.470328 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:28.469909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43281 (local address 127.0.98.1:33501)
0924 04:16:28.470179 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:28.470234 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":86,"threads_started":1}
I20260924 04:16:28.607854 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:28.607451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43281 (local address 127.0.98.2:35875)
0924 04:16:28.607691 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:28.607752 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":72,"threads_started":1}
I20260924 04:16:28.790587 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:28.790149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43281 (local address 127.0.98.3:49605)
0924 04:16:28.790421 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:28.790481 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":72,"threads_started":1}
I20260924 04:16:29.324736 15733 raft_consensus.cc:482] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:16:29.324865 15733 raft_consensus.cc:504] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f4ec3e52e1a494c82be7558bea4ff26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36613 } } peers { permanent_uuid: "683ba6100ecc4be5a17b03622dcad539" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 } } peers { permanent_uuid: "ca07c3fe6ef6478c97f25cfb38c349ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34429 } }
I20260924 04:16:29.325260 15733 leader_election.cc:288] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f4ec3e52e1a494c82be7558bea4ff26 (127.0.98.3:36613), 683ba6100ecc4be5a17b03622dcad539 (127.0.98.2:34529)
W20260924 04:16:29.325974 15312 leader_election.cc:341] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f4ec3e52e1a494c82be7558bea4ff26 (127.0.98.3:36613): Not found: Tablet not found: 2e3be8cfaa9b4c2c8aeb33b3fd49d74c
W20260924 04:16:29.326082 15313 leader_election.cc:341] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 683ba6100ecc4be5a17b03622dcad539 (127.0.98.2:34529): Not found: Tablet not found: 2e3be8cfaa9b4c2c8aeb33b3fd49d74c
I20260924 04:16:29.326176 15313 leader_election.cc:302] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca07c3fe6ef6478c97f25cfb38c349ac; no voters: 3f4ec3e52e1a494c82be7558bea4ff26, 683ba6100ecc4be5a17b03622dcad539
I20260924 04:16:29.326318 15733 raft_consensus.cc:2726] T 2e3be8cfaa9b4c2c8aeb33b3fd49d74c P ca07c3fe6ef6478c97f25cfb38c349ac [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 04:16:29.471558 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:29.471049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43281 (local address 127.0.98.1:59585)
0924 04:16:29.471376 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:29.471436 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":88,"threads_started":1}
I20260924 04:16:29.608953 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:29.608553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43281 (local address 127.0.98.2:53829)
0924 04:16:29.608798 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:29.608855 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":72,"threads_started":1}
I20260924 04:16:29.791697 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:29.791304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43281 (local address 127.0.98.3:58997)
0924 04:16:29.791542 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:29.791596 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":73,"threads_started":1}
I20260924 04:16:30.472703 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:30.472281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43281 (local address 127.0.98.1:48107)
0924 04:16:30.472547 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:30.472605 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":83,"threads_started":1}
I20260924 04:16:30.610018 15738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:30.609628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43281 (local address 127.0.98.2:36307)
0924 04:16:30.609869 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:30.609924 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":70,"threads_started":1}
I20260924 04:16:30.792757 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:30.792393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43281 (local address 127.0.98.3:36821)
0924 04:16:30.792612 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:30.792663 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":69,"threads_started":1}
I20260924 04:16:31.474005 15740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:31.473539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43281 (local address 127.0.98.1:48151)
0924 04:16:31.473829 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:31.473894 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":101,"threads_started":1}
I20260924 04:16:31.611074 15741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:31.610673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43281 (local address 127.0.98.2:40265)
0924 04:16:31.610925 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:31.610982 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":75,"threads_started":1}
I20260924 04:16:31.793852 15742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:31.793459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43281 (local address 127.0.98.3:55923)
0924 04:16:31.793699 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:31.793759 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":72,"threads_started":1}
I20260924 04:16:32.475181 15743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:32.474735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43281 (local address 127.0.98.1:34835)
0924 04:16:32.475022 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:32.475086 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":77,"threads_started":1}
I20260924 04:16:32.612264 15744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:32.611858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43281 (local address 127.0.98.2:33399)
0924 04:16:32.612105 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:32.612165 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":72,"threads_started":1}
I20260924 04:16:32.794957 15745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:32.794523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43281 (local address 127.0.98.3:49275)
0924 04:16:32.794776 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:32.794863 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":73,"threads_started":1}
I20260924 04:16:33.476541 15746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:33.476115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43281 (local address 127.0.98.1:48527)
0924 04:16:33.476353 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:33.476440 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":69,"threads_started":1}
I20260924 04:16:33.613559 15747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:33.613149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43281 (local address 127.0.98.2:47979)
0924 04:16:33.613394 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:33.613458 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":74,"threads_started":1}
I20260924 04:16:33.792680 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 15262
I20260924 04:16:33.796260 15748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:33.795861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43281 (local address 127.0.98.3:50745)
0924 04:16:33.796107 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:33.796166 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":67,"threads_started":1}
I20260924 04:16:33.797737 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 15429
I20260924 04:16:33.802310 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 15563
2026-09-24T04:16:33Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19208 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:16:33 dist-test-slave-p401 krb5kdc[15751](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:16:33 dist-test-slave-p401 krb5kdc[15751](info): set up 2 sockets
Sep 24 04:16:33 dist-test-slave-p401 krb5kdc[15751](info): commencing operation
krb5kdc: starting...
W20260924 04:16:34.771214 392 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/test-user.keytab.
Sep 24 04:16:34 dist-test-slave-p401 krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:16:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:16:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:16:34.898651 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42685
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42685
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:16:35.008965 15764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:16:35.009249 15764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:16:35.009300 15764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:16:35.013393 15764 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 04:16:35.013458 15764 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 04:16:35.013474 15764 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 04:16:35.013486 15764 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:16:35.013504 15764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:16:35.013520 15764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:16:35.013541 15764 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:16:35.013566 15764 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:16:35.018985 15764 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42685
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:16:35.020104 15764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:16:35.021173 15764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:16:35.030025 15772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:16:35.030184 15764 server_base.cc:1034] running on GCE node
W20260924 04:16:35.030072 15770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:16:35.030107 15769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:16:35.030987 15764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:16:35.032109 15764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:16:35.033277 15764 hybrid_clock.cc:648] HybridClock initialized: now 1790223395033256 us; error 33 us; skew 500 ppm
Sep 24 04:16:35 dist-test-slave-p401 krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:16:35.035989 15764 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:16:35.036878 15764 webserver.cc:460] Webserver started at http://127.0.98.62:41215/ using document root <none> and password file <none>
I20260924 04:16:35.037407 15764 fs_manager.cc:359] Metadata directory not provided
I20260924 04:16:35.037473 15764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:16:35.037657 15764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:16:35.039722 15764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/master-0/data/instance:
uuid: "a03a0021d8a04239b1acb304a66797ac"
format_stamp: "Formatted at 2026-09-24 04:16:35 on dist-test-slave-p401"
server_key: "7118c983872934d3829802ecad811fa3"
server_key_iv: "2fd615e22e241d5dfaedc89b258fdeb9"
server_key_version: "encryptionkey@0"
I20260924 04:16:35.040266 15764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/master-0/wal/instance:
uuid: "a03a0021d8a04239b1acb304a66797ac"
format_stamp: "Formatted at 2026-09-24 04:16:35 on dist-test-slave-p401"
server_key: "7118c983872934d3829802ecad811fa3"
server_key_iv: "2fd615e22e241d5dfaedc89b258fdeb9"
server_key_version: "encryptionkey@0"
I20260924 04:16:35.043884 15764 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 04:16:35.046267 15779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:16:35.047556 15764 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260924 04:16:35.047668 15764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/master-0/wal
uuid: "a03a0021d8a04239b1acb304a66797ac"
format_stamp: "Formatted at 2026-09-24 04:16:35 on dist-test-slave-p401"
server_key: "7118c983872934d3829802ecad811fa3"
server_key_iv: "2fd615e22e241d5dfaedc89b258fdeb9"
server_key_version: "encryptionkey@0"
I20260924 04:16:35.047758 15764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:16:35.062052 15764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:16:35.065100 15764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:16:35.065271 15764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:16:35.072014 15764 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42685
I20260924 04:16:35.072270 15831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42685 every 8 connection(s)
I20260924 04:16:35.072938 15764 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:16:35.074919 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 15764
I20260924 04:16:35.075040 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/master-0/wal/instance
I20260924 04:16:35.075371 392 external_mini_cluster.cc:1428] Setting key 5b32e3a9ad031ef9a8b228c687ab3589
I20260924 04:16:35.075953 15832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:16:35.082577 15832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac: Bootstrap starting.
Sep 24 04:16:35 dist-test-slave-p401 krb5kdc[15751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:16:35.085022 15832 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac: Neither blocks nor log segments found. Creating new log.
I20260924 04:16:35.085862 15832 log.cc:824] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac: Log is configured to *not* fsync() on all Append() calls
I20260924 04:16:35.088008 15832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac: No bootstrap required, opened a new log
I20260924 04:16:35.089622 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:35.076645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33548 (local address 127.0.98.62:42685)
0924 04:16:35.077045 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:35.077064 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:35.077105 (+ 41us) server_negotiation.cc:407] Connection header received
0924 04:16:35.077935 (+ 830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:35.077958 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:35.078387 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:35.078753 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:16:35.079402 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:35.080886 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:35.081700 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:35.082361 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:35.082521 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:35.084893 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:16:35.084899 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:16:35.084915 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:16:35.084945 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:16:35.086622 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:35.087090 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:35.087096 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:35.087101 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:35.087253 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:35.087506 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:35.087509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:35.087510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:35.087995 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:16:35.088184 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:35.088451 (+ 267us) server_negotiation.cc:299] Negotiation successful
0924 04:16:35.088730 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":118,"threads_started":1}
I20260924 04:16:35.092098 15832 raft_consensus.cc:348] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a03a0021d8a04239b1acb304a66797ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42685 } }
I20260924 04:16:35.092316 15832 raft_consensus.cc:374] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:16:35.092356 15832 raft_consensus.cc:729] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a03a0021d8a04239b1acb304a66797ac, State: Initialized, Role: FOLLOWER
I20260924 04:16:35.092722 15832 consensus_queue.cc:260] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a03a0021d8a04239b1acb304a66797ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42685 } }
I20260924 04:16:35.092841 15832 raft_consensus.cc:388] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:16:35.092885 15832 raft_consensus.cc:482] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:16:35.092952 15832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:16:35.093735 15832 raft_consensus.cc:504] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a03a0021d8a04239b1acb304a66797ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42685 } }
I20260924 04:16:35.094020 15832 leader_election.cc:302] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a03a0021d8a04239b1acb304a66797ac; no voters:
I20260924 04:16:35.094254 15832 leader_election.cc:288] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:16:35.094352 15837 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:16:35.094564 15837 raft_consensus.cc:686] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac [term 1 LEADER]: Becoming Leader. State: Replica: a03a0021d8a04239b1acb304a66797ac, State: Running, Role: LEADER
I20260924 04:16:35.094823 15837 consensus_queue.cc:237] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a03a0021d8a04239b1acb304a66797ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42685 } }
I20260924 04:16:35.095409 15832 sys_catalog.cc:564] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:16:35.096566 15838 sys_catalog.cc:455] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a03a0021d8a04239b1acb304a66797ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a03a0021d8a04239b1acb304a66797ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42685 } } }
I20260924 04:16:35.096571 15839 sys_catalog.cc:455] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader a03a0021d8a04239b1acb304a66797ac. Latest consensus state: current_term: 1 leader_uuid: "a03a0021d8a04239b1acb304a66797ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a03a0021d8a04239b1acb304a66797ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42685 } } }
I20260924 04:16:35.096697 15839 sys_catalog.cc:458] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac [sys.catalog]: This master's current role is: LEADER
I20260924 04:16:35.096697 15838 sys_catalog.cc:458] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac [sys.catalog]: This master's current role is: LEADER
I20260924 04:16:35.097674 15846 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:16:35.100500 15846 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:16:35.105984 15846 catalog_manager.cc:1347] Generated new cluster ID: 7e4a25d231f7467a98226bbbb4df0995
I20260924 04:16:35.106051 15846 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:16:35.134737 15846 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:16:35.135779 15846 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:16:35.139858 15846 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac: Generated new TSK 0
I20260924 04:16:35.140511 15846 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:16:35.172626 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42685
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:16:35.279397 15858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:16:35.279652 15858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:16:35.279693 15858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:16:35.283716 15858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:16:35.283778 15858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:16:35.283843 15858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:16:35.289086 15858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:16:35.290118 15858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:16:35.291111 15858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:16:35.300571 15863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:16:35.300642 15858 server_base.cc:1034] running on GCE node
W20260924 04:16:35.300572 15866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:16:35.300586 15864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:16:35.301203 15858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:16:35.301800 15858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:16:35.302969 15858 hybrid_clock.cc:648] HybridClock initialized: now 1790223395302951 us; error 35 us; skew 500 ppm
Sep 24 04:16:35 dist-test-slave-p401 krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:16:35.305557 15858 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:16:35.306394 15858 webserver.cc:460] Webserver started at http://127.0.98.1:44351/ using document root <none> and password file <none>
I20260924 04:16:35.306916 15858 fs_manager.cc:359] Metadata directory not provided
I20260924 04:16:35.306978 15858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:16:35.307157 15858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:16:35.309161 15858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-0/data/instance:
uuid: "7a632792b0b5464bb290c8420560401d"
format_stamp: "Formatted at 2026-09-24 04:16:35 on dist-test-slave-p401"
server_key: "e6425b62388f47ff1a883d17ffe9191d"
server_key_iv: "dc187a9f40f73ced6e2a5aed30e12197"
server_key_version: "encryptionkey@0"
I20260924 04:16:35.309684 15858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7a632792b0b5464bb290c8420560401d"
format_stamp: "Formatted at 2026-09-24 04:16:35 on dist-test-slave-p401"
server_key: "e6425b62388f47ff1a883d17ffe9191d"
server_key_iv: "dc187a9f40f73ced6e2a5aed30e12197"
server_key_version: "encryptionkey@0"
I20260924 04:16:35.313237 15858 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:16:35.315487 15873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:16:35.316646 15858 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 04:16:35.316744 15858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-0/wal
uuid: "7a632792b0b5464bb290c8420560401d"
format_stamp: "Formatted at 2026-09-24 04:16:35 on dist-test-slave-p401"
server_key: "e6425b62388f47ff1a883d17ffe9191d"
server_key_iv: "dc187a9f40f73ced6e2a5aed30e12197"
server_key_version: "encryptionkey@0"
I20260924 04:16:35.316835 15858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:16:35.325995 15858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:16:35.328739 15858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:16:35.328907 15858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:16:35.329440 15858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:16:35.330252 15858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:16:35.330299 15858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:16:35.330360 15858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:16:35.330386 15858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:16:35.338581 15858 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37857
I20260924 04:16:35.338604 15986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37857 every 8 connection(s)
I20260924 04:16:35.339440 15858 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 04:16:35 dist-test-slave-p401 krb5kdc[15751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:16:35.348415 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 15858
I20260924 04:16:35.348549 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:16:35.348841 392 external_mini_cluster.cc:1428] Setting key cc68714812a56dd530a2173dd5c33337
I20260924 04:16:35.352317 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:35.341179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47577 (local address 127.0.98.62:42685)
0924 04:16:35.341295 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:35.341299 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:35.342280 (+ 981us) server_negotiation.cc:407] Connection header received
0924 04:16:35.343104 (+ 824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:35.343107 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:35.343151 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:35.343221 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:16:35.344307 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:35.344866 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:35.345917 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:35.346133 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:35.346199 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:35.348594 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:16:35.348598 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:16:35.348601 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:16:35.348628 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:16:35.350001 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:35.350570 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:35.350573 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:35.350574 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:35.350660 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:35.350996 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:35.350998 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:35.350999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:35.351212 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:16:35.351309 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:35.351930 (+ 621us) server_negotiation.cc:299] Negotiation successful
0924 04:16:35.352082 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 04:16:35.352989 15989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:35.341520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.1:47577)
0924 04:16:35.342144 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:35.342183 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 04:16:35.342930 (+ 747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:16:35.343403 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:16:35.343410 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:16:35.343726 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:16:35.344177 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:35.344191 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:35.344963 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:35.344966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:35.345828 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:35.345837 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:35.346234 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:35.346236 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:35.346390 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:35.346858 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:16:35.346874 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:16:35.348393 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:16:35.350107 (+ 1714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:35.350112 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:35.350125 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:35.350450 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:35.350740 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:35.350743 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:35.350744 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:35.350914 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:35.351308 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:16:35.351314 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:16:35.351636 (+ 322us) client_negotiation.cc:769] Sending connection context
0924 04:16:35.351858 (+ 222us) client_negotiation.cc:240] Negotiation successful
0924 04:16:35.352127 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":105,"threads_started":1}
I20260924 04:16:35.354243 15987 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42685
I20260924 04:16:35.354525 15987 heartbeater.cc:464] Registering TS with master...
I20260924 04:16:35.355126 15987 heartbeater.cc:511] Master 127.0.98.62:42685 requested a full tablet report, sending...
I20260924 04:16:35.356787 15796 ts_manager.cc:194] Registered new tserver with Master: 7a632792b0b5464bb290c8420560401d (127.0.98.1:37857)
I20260924 04:16:35.357918 15796 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47577
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:16:35.373991 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42685
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:16:35.482816 15992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:16:35.483103 15992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:16:35.483144 15992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:16:35.487106 15992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:16:35.487170 15992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:16:35.487248 15992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:16:35.492437 15992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:16:35.493445 15992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:16:35.494418 15992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:16:35.503997 15997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:16:35.504009 16000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:16:35.504156 15992 server_base.cc:1034] running on GCE node
W20260924 04:16:35.504000 15998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:16:35.504659 15992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:16:35.505296 15992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:16:35.506461 15992 hybrid_clock.cc:648] HybridClock initialized: now 1790223395506439 us; error 37 us; skew 500 ppm
Sep 24 04:16:35 dist-test-slave-p401 krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:16:35.509049 15992 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:16:35.509930 15992 webserver.cc:460] Webserver started at http://127.0.98.2:45289/ using document root <none> and password file <none>
I20260924 04:16:35.510452 15992 fs_manager.cc:359] Metadata directory not provided
I20260924 04:16:35.510516 15992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:16:35.510712 15992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:16:35.512768 15992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-1/data/instance:
uuid: "adfc52788f744448b30368cba1c1cd5e"
format_stamp: "Formatted at 2026-09-24 04:16:35 on dist-test-slave-p401"
server_key: "7c2671fb1614616197695f60d2825b1e"
server_key_iv: "9ee2272ba04d60a708429de355b63914"
server_key_version: "encryptionkey@0"
I20260924 04:16:35.513298 15992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-1/wal/instance:
uuid: "adfc52788f744448b30368cba1c1cd5e"
format_stamp: "Formatted at 2026-09-24 04:16:35 on dist-test-slave-p401"
server_key: "7c2671fb1614616197695f60d2825b1e"
server_key_iv: "9ee2272ba04d60a708429de355b63914"
server_key_version: "encryptionkey@0"
I20260924 04:16:35.516973 15992 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260924 04:16:35.519523 16007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:16:35.520744 15992 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 04:16:35.520877 15992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-1/wal
uuid: "adfc52788f744448b30368cba1c1cd5e"
format_stamp: "Formatted at 2026-09-24 04:16:35 on dist-test-slave-p401"
server_key: "7c2671fb1614616197695f60d2825b1e"
server_key_iv: "9ee2272ba04d60a708429de355b63914"
server_key_version: "encryptionkey@0"
I20260924 04:16:35.520979 15992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:16:35.537871 15992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:16:35.540747 15992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:16:35.540936 15992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:16:35.541512 15992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:16:35.542302 15992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:16:35.542348 15992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:16:35.542399 15992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:16:35.542424 15992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:16:35.550714 15992 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44431
I20260924 04:16:35.550756 16120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44431 every 8 connection(s)
I20260924 04:16:35.551697 15992 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:16:35.560132 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 15992
I20260924 04:16:35.560230 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-1/wal/instance
Sep 24 04:16:35 dist-test-slave-p401 krb5kdc[15751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:16:35.560497 392 external_mini_cluster.cc:1428] Setting key 560c5bd13c3e4b4bbd43754af8a87134
I20260924 04:16:35.564462 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:35.553400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53179 (local address 127.0.98.62:42685)
0924 04:16:35.553536 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:35.553539 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:35.554532 (+ 993us) server_negotiation.cc:407] Connection header received
0924 04:16:35.555497 (+ 965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:35.555500 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:35.555551 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:35.555623 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:16:35.556770 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:35.557307 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:35.558345 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:35.558559 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:35.558626 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:35.561104 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:16:35.561108 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:16:35.561110 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:16:35.561135 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:16:35.562409 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:35.562952 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:35.562954 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:35.562956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:35.563019 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:35.563356 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:35.563358 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:35.563359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:35.563591 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:16:35.563685 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:35.564171 (+ 486us) server_negotiation.cc:299] Negotiation successful
0924 04:16:35.564289 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 04:16:35.565145 16123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:35.553770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.2:53179)
0924 04:16:35.554396 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:35.554437 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 04:16:35.555281 (+ 844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:16:35.555823 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:16:35.555830 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:16:35.556175 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:16:35.556642 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:35.556656 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:35.557400 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:35.557404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:35.558260 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:35.558268 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:35.558669 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:35.558672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:35.558826 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:35.559352 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:16:35.559367 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:16:35.560948 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:16:35.562525 (+ 1577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:35.562531 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:35.562546 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:35.562873 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:35.563085 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:35.563087 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:35.563089 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:35.563279 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:35.563686 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:16:35.563691 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:16:35.563949 (+ 258us) client_negotiation.cc:769] Sending connection context
0924 04:16:35.564096 (+ 147us) client_negotiation.cc:240] Negotiation successful
0924 04:16:35.564342 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":121,"threads_started":1}
I20260924 04:16:35.566260 16121 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42685
I20260924 04:16:35.566516 16121 heartbeater.cc:464] Registering TS with master...
I20260924 04:16:35.567108 16121 heartbeater.cc:511] Master 127.0.98.62:42685 requested a full tablet report, sending...
I20260924 04:16:35.568274 15796 ts_manager.cc:194] Registered new tserver with Master: adfc52788f744448b30368cba1c1cd5e (127.0.98.2:44431)
I20260924 04:16:35.569024 15796 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53179
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:16:35.586155 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42685
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:16:35.693915 16126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:16:35.694164 16126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:16:35.694218 16126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:16:35.698318 16126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:16:35.698372 16126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:16:35.698449 16126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:16:35.703764 16126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:16:35.704741 16126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:16:35.705729 16126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:16:35.714862 16131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:16:35.714920 16132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:16:35.714867 16134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:16:35.715085 16126 server_base.cc:1034] running on GCE node
I20260924 04:16:35.715580 16126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:16:35.716163 16126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:16:35.717325 16126 hybrid_clock.cc:648] HybridClock initialized: now 1790223395717305 us; error 32 us; skew 500 ppm
Sep 24 04:16:35 dist-test-slave-p401 krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:16:35.720006 16126 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:16:35.720912 16126 webserver.cc:460] Webserver started at http://127.0.98.3:40435/ using document root <none> and password file <none>
I20260924 04:16:35.721477 16126 fs_manager.cc:359] Metadata directory not provided
I20260924 04:16:35.721542 16126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:16:35.721731 16126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:16:35.723765 16126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-2/data/instance:
uuid: "d8b0a26a9a4141fb95452f16af8283fc"
format_stamp: "Formatted at 2026-09-24 04:16:35 on dist-test-slave-p401"
server_key: "f7ae8f9b974f75cd6d051bef2100e1d5"
server_key_iv: "a515fd50bab75fbdfdd9f322ea1a8b83"
server_key_version: "encryptionkey@0"
I20260924 04:16:35.724294 16126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d8b0a26a9a4141fb95452f16af8283fc"
format_stamp: "Formatted at 2026-09-24 04:16:35 on dist-test-slave-p401"
server_key: "f7ae8f9b974f75cd6d051bef2100e1d5"
server_key_iv: "a515fd50bab75fbdfdd9f322ea1a8b83"
server_key_version: "encryptionkey@0"
I20260924 04:16:35.727946 16126 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:16:35.730315 16141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:16:35.731606 16126 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:16:35.731707 16126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-2/wal
uuid: "d8b0a26a9a4141fb95452f16af8283fc"
format_stamp: "Formatted at 2026-09-24 04:16:35 on dist-test-slave-p401"
server_key: "f7ae8f9b974f75cd6d051bef2100e1d5"
server_key_iv: "a515fd50bab75fbdfdd9f322ea1a8b83"
server_key_version: "encryptionkey@0"
I20260924 04:16:35.731799 16126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:16:35.751169 16126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:16:35.753903 16126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:16:35.754069 16126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:16:35.754607 16126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:16:35.755395 16126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:16:35.755440 16126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:16:35.755491 16126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:16:35.755518 16126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:16:35.763696 16126 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37753
I20260924 04:16:35.763727 16254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37753 every 8 connection(s)
I20260924 04:16:35.764529 16126 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:16:35.771870 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 16126
I20260924 04:16:35.771972 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:16:35.772212 392 external_mini_cluster.cc:1428] Setting key dd84a5b1bd655fe7472f31c50b2acbff
Sep 24 04:16:35 dist-test-slave-p401 krb5kdc[15751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:16:35.776821 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:35.766260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55673 (local address 127.0.98.62:42685)
0924 04:16:35.766401 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:35.766406 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:35.767500 (+ 1094us) server_negotiation.cc:407] Connection header received
0924 04:16:35.768335 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:35.768353 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:35.768399 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:35.768470 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:16:35.769609 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:35.770129 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:35.771210 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:35.771477 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:35.771560 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:35.773984 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:16:35.773987 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:16:35.773990 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:16:35.774017 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:16:35.775175 (+ 1158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:35.775649 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:35.775651 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:35.775653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:35.775719 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:35.775965 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:35.775966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:35.775968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:35.776123 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:16:35.776190 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:35.776616 (+ 426us) server_negotiation.cc:299] Negotiation successful
0924 04:16:35.776707 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 04:16:35.777627 16257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:35.766642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.3:55673)
0924 04:16:35.767344 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:35.767387 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 04:16:35.768156 (+ 769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:16:35.768617 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:16:35.768623 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:16:35.768992 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:16:35.769482 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:35.769496 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:35.770244 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:35.770247 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:35.771123 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:35.771132 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:35.771606 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:35.771609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:35.771746 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:35.772339 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:16:35.772356 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:16:35.773828 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:16:35.775321 (+ 1493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:35.775326 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:35.775337 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:35.775571 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:35.775779 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:35.775781 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:35.775782 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:35.775905 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:35.776202 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:16:35.776207 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:16:35.776437 (+ 230us) client_negotiation.cc:769] Sending connection context
0924 04:16:35.776585 (+ 148us) client_negotiation.cc:240] Negotiation successful
0924 04:16:35.776862 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":116,"threads_started":1}
I20260924 04:16:35.778658 16255 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42685
I20260924 04:16:35.778867 16255 heartbeater.cc:464] Registering TS with master...
I20260924 04:16:35.779333 16255 heartbeater.cc:511] Master 127.0.98.62:42685 requested a full tablet report, sending...
I20260924 04:16:35.780297 15796 ts_manager.cc:194] Registered new tserver with Master: d8b0a26a9a4141fb95452f16af8283fc (127.0.98.3:37753)
I20260924 04:16:35.780853 15796 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55673
I20260924 04:16:35.785750 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:16:35.793919 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:35.787487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33560 (local address 127.0.98.62:42685)
0924 04:16:35.787611 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:35.787615 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:35.787778 (+ 163us) server_negotiation.cc:407] Connection header received
0924 04:16:35.787865 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:35.787867 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:35.787907 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:35.787988 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:16:35.788589 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:35.789095 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:35.789912 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:35.790133 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:35.790205 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:35.791445 (+ 1240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:16:35.791448 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:16:35.791450 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:16:35.791471 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:16:35.792664 (+ 1193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:35.792999 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:35.793001 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:35.793003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:35.793069 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:35.793245 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:35.793247 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:35.793248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:35.793439 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:16:35.793533 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:35.793675 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 04:16:35.793759 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 04:16:35.796576 15796 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33560:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:16:35.799661 15796 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:16:35.813447 16268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:35.807632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35952 (local address 127.0.98.3:37753)
0924 04:16:35.808306 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:35.808318 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:35.808690 (+ 372us) server_negotiation.cc:407] Connection header received
0924 04:16:35.808891 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:35.808899 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:35.809106 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:35.809240 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:16:35.809497 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:35.810498 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:35.811993 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:35.812747 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:35.812854 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:35.812999 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:35.813195 (+ 196us) server_negotiation.cc:299] Negotiation successful
0924 04:16:35.813290 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":133,"threads_started":1}
I20260924 04:16:35.813619 16271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:35.808273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48352 (local address 127.0.98.1:37857)
0924 04:16:35.808978 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:35.808985 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:35.809003 (+ 18us) server_negotiation.cc:407] Connection header received
0924 04:16:35.809072 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:35.809077 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:35.809252 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:35.809395 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:16:35.809860 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:35.810916 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:35.812444 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:35.813035 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:35.813142 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:35.813269 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:35.813397 (+ 128us) server_negotiation.cc:299] Negotiation successful
0924 04:16:35.813493 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":132,"threads_started":1}
I20260924 04:16:35.813619 16270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:35.808251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36360 (local address 127.0.98.2:44431)
0924 04:16:35.808804 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:35.808815 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:35.809050 (+ 235us) server_negotiation.cc:407] Connection header received
0924 04:16:35.809139 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:35.809143 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:35.809281 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:35.809395 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:16:35.809864 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:35.810601 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:35.812171 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:35.812928 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:35.813015 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:35.813127 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:35.813409 (+ 282us) server_negotiation.cc:299] Negotiation successful
0924 04:16:35.813494 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":142,"threads_started":1}
I20260924 04:16:35.813612 16269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:35.807631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37857 (local address 127.0.0.1:48352)
0924 04:16:35.808689 (+ 1058us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:35.808702 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:16:35.808782 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:16:35.809548 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:16:35.809550 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:16:35.809580 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:16:35.809728 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:35.809735 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:35.811046 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:35.811049 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:35.812336 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:35.812343 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:35.813150 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:35.813154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:35.813250 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:35.813275 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 04:16:35.813355 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 04:16:35.813471 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1001,"mutex_wait_us":135,"spinlock_wait_cycles":2560,"thread_start_us":107,"threads_started":1}
I20260924 04:16:35.814010 16267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:35.807858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44431 (local address 127.0.0.1:36360)
0924 04:16:35.808958 (+ 1100us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:35.808969 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:16:35.809046 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:16:35.809548 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:16:35.809550 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:16:35.809580 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:16:35.809745 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:35.809751 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:35.810756 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:35.810759 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:35.812044 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:35.812054 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:35.813174 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:35.813177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:35.813275 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:35.813293 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 04:16:35.813808 (+ 515us) client_negotiation.cc:240] Negotiation successful
0924 04:16:35.813887 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1048,"mutex_wait_us":97,"spinlock_wait_cycles":6784,"thread_start_us":83,"threads_started":1}
I20260924 04:16:35.814286 16266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:35.807494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37753 (local address 127.0.0.1:35952)
0924 04:16:35.808549 (+ 1055us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:35.808596 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 04:16:35.808735 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:16:35.809243 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:16:35.809249 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:16:35.809269 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:16:35.809398 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:35.809405 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:35.810625 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:35.810629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:35.811887 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:35.811897 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:35.812865 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:35.812869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:35.812984 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:35.813026 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 04:16:35.813983 (+ 957us) client_negotiation.cc:240] Negotiation successful
0924 04:16:35.814061 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":892,"mutex_wait_us":61,"thread_start_us":74,"threads_started":1}
I20260924 04:16:35.816565 15921 tablet_service.cc:1467] Processing CreateTablet for tablet 158d6801864e400298dcae8ad50965ad (DEFAULT_TABLE table=test-table [id=fb16417e3901416ca2309c0b5308b7d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:16:35.816768 16189 tablet_service.cc:1467] Processing CreateTablet for tablet 158d6801864e400298dcae8ad50965ad (DEFAULT_TABLE table=test-table [id=fb16417e3901416ca2309c0b5308b7d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:16:35.816768 16055 tablet_service.cc:1467] Processing CreateTablet for tablet 158d6801864e400298dcae8ad50965ad (DEFAULT_TABLE table=test-table [id=fb16417e3901416ca2309c0b5308b7d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:16:35.817580 15921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 158d6801864e400298dcae8ad50965ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:16:35.817761 16055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 158d6801864e400298dcae8ad50965ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:16:35.817767 16189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 158d6801864e400298dcae8ad50965ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:16:35.823589 16272 tablet_bootstrap.cc:492] T 158d6801864e400298dcae8ad50965ad P 7a632792b0b5464bb290c8420560401d: Bootstrap starting.
I20260924 04:16:35.825378 16274 tablet_bootstrap.cc:492] T 158d6801864e400298dcae8ad50965ad P adfc52788f744448b30368cba1c1cd5e: Bootstrap starting.
I20260924 04:16:35.826076 16272 tablet_bootstrap.cc:654] T 158d6801864e400298dcae8ad50965ad P 7a632792b0b5464bb290c8420560401d: Neither blocks nor log segments found. Creating new log.
I20260924 04:16:35.827126 16272 log.cc:824] T 158d6801864e400298dcae8ad50965ad P 7a632792b0b5464bb290c8420560401d: Log is configured to *not* fsync() on all Append() calls
I20260924 04:16:35.827400 16274 tablet_bootstrap.cc:654] T 158d6801864e400298dcae8ad50965ad P adfc52788f744448b30368cba1c1cd5e: Neither blocks nor log segments found. Creating new log.
I20260924 04:16:35.827737 16273 tablet_bootstrap.cc:492] T 158d6801864e400298dcae8ad50965ad P d8b0a26a9a4141fb95452f16af8283fc: Bootstrap starting.
I20260924 04:16:35.828227 16274 log.cc:824] T 158d6801864e400298dcae8ad50965ad P adfc52788f744448b30368cba1c1cd5e: Log is configured to *not* fsync() on all Append() calls
I20260924 04:16:35.829048 16272 tablet_bootstrap.cc:492] T 158d6801864e400298dcae8ad50965ad P 7a632792b0b5464bb290c8420560401d: No bootstrap required, opened a new log
I20260924 04:16:35.829238 16272 ts_tablet_manager.cc:1397] T 158d6801864e400298dcae8ad50965ad P 7a632792b0b5464bb290c8420560401d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:16:35.829490 16273 tablet_bootstrap.cc:654] T 158d6801864e400298dcae8ad50965ad P d8b0a26a9a4141fb95452f16af8283fc: Neither blocks nor log segments found. Creating new log.
I20260924 04:16:35.830121 16274 tablet_bootstrap.cc:492] T 158d6801864e400298dcae8ad50965ad P adfc52788f744448b30368cba1c1cd5e: No bootstrap required, opened a new log
I20260924 04:16:35.830371 16274 ts_tablet_manager.cc:1397] T 158d6801864e400298dcae8ad50965ad P adfc52788f744448b30368cba1c1cd5e: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260924 04:16:35.830572 16273 log.cc:824] T 158d6801864e400298dcae8ad50965ad P d8b0a26a9a4141fb95452f16af8283fc: Log is configured to *not* fsync() on all Append() calls
I20260924 04:16:35.832121 16273 tablet_bootstrap.cc:492] T 158d6801864e400298dcae8ad50965ad P d8b0a26a9a4141fb95452f16af8283fc: No bootstrap required, opened a new log
I20260924 04:16:35.832281 16273 ts_tablet_manager.cc:1397] T 158d6801864e400298dcae8ad50965ad P d8b0a26a9a4141fb95452f16af8283fc: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 04:16:35.833055 16272 raft_consensus.cc:348] T 158d6801864e400298dcae8ad50965ad P 7a632792b0b5464bb290c8420560401d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b0a26a9a4141fb95452f16af8283fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } } peers { permanent_uuid: "7a632792b0b5464bb290c8420560401d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37857 } } peers { permanent_uuid: "adfc52788f744448b30368cba1c1cd5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44431 } }
I20260924 04:16:35.833312 16272 raft_consensus.cc:374] T 158d6801864e400298dcae8ad50965ad P 7a632792b0b5464bb290c8420560401d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:16:35.833391 16272 raft_consensus.cc:729] T 158d6801864e400298dcae8ad50965ad P 7a632792b0b5464bb290c8420560401d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a632792b0b5464bb290c8420560401d, State: Initialized, Role: FOLLOWER
I20260924 04:16:35.833837 16272 consensus_queue.cc:260] T 158d6801864e400298dcae8ad50965ad P 7a632792b0b5464bb290c8420560401d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b0a26a9a4141fb95452f16af8283fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } } peers { permanent_uuid: "7a632792b0b5464bb290c8420560401d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37857 } } peers { permanent_uuid: "adfc52788f744448b30368cba1c1cd5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44431 } }
I20260924 04:16:35.834024 16274 raft_consensus.cc:348] T 158d6801864e400298dcae8ad50965ad P adfc52788f744448b30368cba1c1cd5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b0a26a9a4141fb95452f16af8283fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } } peers { permanent_uuid: "7a632792b0b5464bb290c8420560401d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37857 } } peers { permanent_uuid: "adfc52788f744448b30368cba1c1cd5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44431 } }
I20260924 04:16:35.834342 16274 raft_consensus.cc:374] T 158d6801864e400298dcae8ad50965ad P adfc52788f744448b30368cba1c1cd5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:16:35.834429 16274 raft_consensus.cc:729] T 158d6801864e400298dcae8ad50965ad P adfc52788f744448b30368cba1c1cd5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adfc52788f744448b30368cba1c1cd5e, State: Initialized, Role: FOLLOWER
I20260924 04:16:35.834522 15987 heartbeater.cc:503] Master 127.0.98.62:42685 was elected leader, sending a full tablet report...
I20260924 04:16:35.834874 16272 ts_tablet_manager.cc:1428] T 158d6801864e400298dcae8ad50965ad P 7a632792b0b5464bb290c8420560401d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 04:16:35.834851 16274 consensus_queue.cc:260] T 158d6801864e400298dcae8ad50965ad P adfc52788f744448b30368cba1c1cd5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b0a26a9a4141fb95452f16af8283fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } } peers { permanent_uuid: "7a632792b0b5464bb290c8420560401d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37857 } } peers { permanent_uuid: "adfc52788f744448b30368cba1c1cd5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44431 } }
I20260924 04:16:35.835485 16121 heartbeater.cc:503] Master 127.0.98.62:42685 was elected leader, sending a full tablet report...
I20260924 04:16:35.836340 16273 raft_consensus.cc:348] T 158d6801864e400298dcae8ad50965ad P d8b0a26a9a4141fb95452f16af8283fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b0a26a9a4141fb95452f16af8283fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } } peers { permanent_uuid: "7a632792b0b5464bb290c8420560401d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37857 } } peers { permanent_uuid: "adfc52788f744448b30368cba1c1cd5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44431 } }
I20260924 04:16:35.836661 16273 raft_consensus.cc:374] T 158d6801864e400298dcae8ad50965ad P d8b0a26a9a4141fb95452f16af8283fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:16:35.836695 16274 ts_tablet_manager.cc:1428] T 158d6801864e400298dcae8ad50965ad P adfc52788f744448b30368cba1c1cd5e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:16:35.836732 16273 raft_consensus.cc:729] T 158d6801864e400298dcae8ad50965ad P d8b0a26a9a4141fb95452f16af8283fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8b0a26a9a4141fb95452f16af8283fc, State: Initialized, Role: FOLLOWER
I20260924 04:16:35.837106 16273 consensus_queue.cc:260] T 158d6801864e400298dcae8ad50965ad P d8b0a26a9a4141fb95452f16af8283fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b0a26a9a4141fb95452f16af8283fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } } peers { permanent_uuid: "7a632792b0b5464bb290c8420560401d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37857 } } peers { permanent_uuid: "adfc52788f744448b30368cba1c1cd5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44431 } }
I20260924 04:16:35.837867 16255 heartbeater.cc:503] Master 127.0.98.62:42685 was elected leader, sending a full tablet report...
I20260924 04:16:35.837927 16273 ts_tablet_manager.cc:1428] T 158d6801864e400298dcae8ad50965ad P d8b0a26a9a4141fb95452f16af8283fc: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260924 04:16:35.840550 15988 tablet.cc:2369] T 158d6801864e400298dcae8ad50965ad P 7a632792b0b5464bb290c8420560401d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:16:35.854389 16278 raft_consensus.cc:482] T 158d6801864e400298dcae8ad50965ad P 7a632792b0b5464bb290c8420560401d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:16:35.854571 16278 raft_consensus.cc:504] T 158d6801864e400298dcae8ad50965ad P 7a632792b0b5464bb290c8420560401d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b0a26a9a4141fb95452f16af8283fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } } peers { permanent_uuid: "7a632792b0b5464bb290c8420560401d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37857 } } peers { permanent_uuid: "adfc52788f744448b30368cba1c1cd5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44431 } }
I20260924 04:16:35.855702 16278 leader_election.cc:288] T 158d6801864e400298dcae8ad50965ad P 7a632792b0b5464bb290c8420560401d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8b0a26a9a4141fb95452f16af8283fc (127.0.98.3:37753), adfc52788f744448b30368cba1c1cd5e (127.0.98.2:44431)
I20260924 04:16:35.859576 16268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:35.856005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49057 (local address 127.0.98.3:37753)
0924 04:16:35.856253 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:35.856258 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:35.856292 (+ 34us) server_negotiation.cc:407] Connection header received
0924 04:16:35.856400 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:35.856402 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:35.856451 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:35.856534 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:16:35.856784 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:35.857570 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:35.858636 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:35.859028 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:35.859104 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:35.859166 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:35.859374 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 04:16:35.859449 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161}
I20260924 04:16:35.859583 16270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:35.856108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60563 (local address 127.0.98.2:44431)
0924 04:16:35.856219 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:35.856223 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:35.856469 (+ 246us) server_negotiation.cc:407] Connection header received
0924 04:16:35.856545 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:35.856547 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:35.856603 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:35.856671 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:16:35.856898 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:35.857593 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:35.858650 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:35.859061 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:35.859155 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:35.859216 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:35.859414 (+ 198us) server_negotiation.cc:299] Negotiation successful
0924 04:16:35.859485 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 04:16:35.859588 16282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:35.856010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44431 (local address 127.0.98.1:60563)
0924 04:16:35.856385 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:35.856397 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:16:35.856476 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:16:35.856693 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:16:35.856695 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:16:35.856708 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:16:35.856813 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:35.856819 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:35.857707 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:35.857710 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:35.858525 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:35.858535 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:35.859167 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:35.859171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:35.859297 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:35.859316 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 04:16:35.859389 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 04:16:35.859449 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":104,"threads_started":1}
I20260924 04:16:35.859594 16281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:35.855929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37753 (local address 127.0.98.1:49057)
0924 04:16:35.856171 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:35.856185 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:16:35.856289 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:16:35.856551 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:16:35.856554 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:16:35.856569 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:16:35.856682 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:35.856688 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:35.857686 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:35.857690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:35.858516 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:35.858524 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:35.859136 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:35.859140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:35.859252 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:35.859277 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 04:16:35.859345 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 04:16:35.859413 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":89,"threads_started":1}
I20260924 04:16:35.860159 16075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "158d6801864e400298dcae8ad50965ad" candidate_uuid: "7a632792b0b5464bb290c8420560401d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adfc52788f744448b30368cba1c1cd5e" is_pre_election: true
I20260924 04:16:35.860188 16209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "158d6801864e400298dcae8ad50965ad" candidate_uuid: "7a632792b0b5464bb290c8420560401d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b0a26a9a4141fb95452f16af8283fc" is_pre_election: true
I20260924 04:16:35.860432 16075 raft_consensus.cc:2454] T 158d6801864e400298dcae8ad50965ad P adfc52788f744448b30368cba1c1cd5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a632792b0b5464bb290c8420560401d in term 0.
I20260924 04:16:35.860445 16209 raft_consensus.cc:2454] T 158d6801864e400298dcae8ad50965ad P d8b0a26a9a4141fb95452f16af8283fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a632792b0b5464bb290c8420560401d in term 0.
I20260924 04:16:35.860864 15875 leader_election.cc:302] T 158d6801864e400298dcae8ad50965ad P 7a632792b0b5464bb290c8420560401d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a632792b0b5464bb290c8420560401d, d8b0a26a9a4141fb95452f16af8283fc; no voters:
I20260924 04:16:35.861109 16278 raft_consensus.cc:2781] T 158d6801864e400298dcae8ad50965ad P 7a632792b0b5464bb290c8420560401d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:16:35.861191 16278 raft_consensus.cc:482] T 158d6801864e400298dcae8ad50965ad P 7a632792b0b5464bb290c8420560401d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:16:35.861222 16278 raft_consensus.cc:3037] T 158d6801864e400298dcae8ad50965ad P 7a632792b0b5464bb290c8420560401d [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:16:35.862206 16278 raft_consensus.cc:504] T 158d6801864e400298dcae8ad50965ad P 7a632792b0b5464bb290c8420560401d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b0a26a9a4141fb95452f16af8283fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } } peers { permanent_uuid: "7a632792b0b5464bb290c8420560401d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37857 } } peers { permanent_uuid: "adfc52788f744448b30368cba1c1cd5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44431 } }
I20260924 04:16:35.862536 16278 leader_election.cc:288] T 158d6801864e400298dcae8ad50965ad P 7a632792b0b5464bb290c8420560401d [CANDIDATE]: Term 1 election: Requested vote from peers d8b0a26a9a4141fb95452f16af8283fc (127.0.98.3:37753), adfc52788f744448b30368cba1c1cd5e (127.0.98.2:44431)
I20260924 04:16:35.862831 16209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "158d6801864e400298dcae8ad50965ad" candidate_uuid: "7a632792b0b5464bb290c8420560401d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b0a26a9a4141fb95452f16af8283fc"
I20260924 04:16:35.862851 16075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "158d6801864e400298dcae8ad50965ad" candidate_uuid: "7a632792b0b5464bb290c8420560401d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adfc52788f744448b30368cba1c1cd5e"
I20260924 04:16:35.862937 16209 raft_consensus.cc:3037] T 158d6801864e400298dcae8ad50965ad P d8b0a26a9a4141fb95452f16af8283fc [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:16:35.862951 16075 raft_consensus.cc:3037] T 158d6801864e400298dcae8ad50965ad P adfc52788f744448b30368cba1c1cd5e [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:16:35.863974 16075 raft_consensus.cc:2454] T 158d6801864e400298dcae8ad50965ad P adfc52788f744448b30368cba1c1cd5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a632792b0b5464bb290c8420560401d in term 1.
I20260924 04:16:35.863993 16209 raft_consensus.cc:2454] T 158d6801864e400298dcae8ad50965ad P d8b0a26a9a4141fb95452f16af8283fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a632792b0b5464bb290c8420560401d in term 1.
I20260924 04:16:35.864253 15877 leader_election.cc:302] T 158d6801864e400298dcae8ad50965ad P 7a632792b0b5464bb290c8420560401d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7a632792b0b5464bb290c8420560401d, adfc52788f744448b30368cba1c1cd5e, d8b0a26a9a4141fb95452f16af8283fc; no voters:
I20260924 04:16:35.864425 16278 raft_consensus.cc:2781] T 158d6801864e400298dcae8ad50965ad P 7a632792b0b5464bb290c8420560401d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:16:35.864650 16278 raft_consensus.cc:686] T 158d6801864e400298dcae8ad50965ad P 7a632792b0b5464bb290c8420560401d [term 1 LEADER]: Becoming Leader. State: Replica: 7a632792b0b5464bb290c8420560401d, State: Running, Role: LEADER
I20260924 04:16:35.864945 16278 consensus_queue.cc:237] T 158d6801864e400298dcae8ad50965ad P 7a632792b0b5464bb290c8420560401d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b0a26a9a4141fb95452f16af8283fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } } peers { permanent_uuid: "7a632792b0b5464bb290c8420560401d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37857 } } peers { permanent_uuid: "adfc52788f744448b30368cba1c1cd5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44431 } }
I20260924 04:16:35.868002 15795 catalog_manager.cc:5495] T 158d6801864e400298dcae8ad50965ad P 7a632792b0b5464bb290c8420560401d reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a632792b0b5464bb290c8420560401d (127.0.98.1). New cstate: current_term: 1 leader_uuid: "7a632792b0b5464bb290c8420560401d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b0a26a9a4141fb95452f16af8283fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a632792b0b5464bb290c8420560401d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "adfc52788f744448b30368cba1c1cd5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44431 } health_report { overall_health: UNKNOWN } } }
I20260924 04:16:35.877630 16271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:35.874601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48366 (local address 127.0.98.1:37857)
0924 04:16:35.874711 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:35.874715 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:35.874726 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:16:35.874879 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:35.874881 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:35.874930 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:35.875011 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:16:35.875193 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:35.875680 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:35.876491 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:35.876692 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:35.876763 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:35.876964 (+ 201us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:16:35.877357 (+ 393us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:16:35.877439 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:35.877466 (+ 27us) server_negotiation.cc:299] Negotiation successful
0924 04:16:35.877509 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 04:16:35.883128 16209 raft_consensus.cc:1261] T 158d6801864e400298dcae8ad50965ad P d8b0a26a9a4141fb95452f16af8283fc [term 1 FOLLOWER]: Refusing update from remote peer 7a632792b0b5464bb290c8420560401d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:16:35.883128 16075 raft_consensus.cc:1261] T 158d6801864e400298dcae8ad50965ad P adfc52788f744448b30368cba1c1cd5e [term 1 FOLLOWER]: Refusing update from remote peer 7a632792b0b5464bb290c8420560401d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:16:35.883814 16278 consensus_queue.cc:1035] T 158d6801864e400298dcae8ad50965ad P 7a632792b0b5464bb290c8420560401d [LEADER]: Connected to new peer: Peer: permanent_uuid: "adfc52788f744448b30368cba1c1cd5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:16:35.884002 16283 consensus_queue.cc:1035] T 158d6801864e400298dcae8ad50965ad P 7a632792b0b5464bb290c8420560401d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b0a26a9a4141fb95452f16af8283fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:16:35.891773 16287 mvcc.cc:205] Tried to move back new op lower bound from 7332755029530644480 to 7332755029465169920. Current Snapshot: MvccSnapshot[applied={T|T < 7332755029530644480}]
I20260924 04:16:35.894634 16288 mvcc.cc:205] Tried to move back new op lower bound from 7332755029530644480 to 7332755029465169920. Current Snapshot: MvccSnapshot[applied={T|T < 7332755029530644480 or (T in {7332755029530644480})}]
I20260924 04:16:35.899304 15795 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:16:35.899480 15795 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:16:35.901746 15795 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac: Sending DeleteTablet for 3 replicas of tablet 158d6801864e400298dcae8ad50965ad
I20260924 04:16:35.902540 16055 tablet_service.cc:1514] Processing DeleteTablet for tablet 158d6801864e400298dcae8ad50965ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:16:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36360
I20260924 04:16:35.902518 15921 tablet_service.cc:1514] Processing DeleteTablet for tablet 158d6801864e400298dcae8ad50965ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:16:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48352
I20260924 04:16:35.902519 16189 tablet_service.cc:1514] Processing DeleteTablet for tablet 158d6801864e400298dcae8ad50965ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:16:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35952
I20260924 04:16:35.902731 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 15858
I20260924 04:16:35.904057 16296 tablet_replica.cc:330] T 158d6801864e400298dcae8ad50965ad P adfc52788f744448b30368cba1c1cd5e: stopping tablet replica
I20260924 04:16:35.904333 16296 raft_consensus.cc:2229] T 158d6801864e400298dcae8ad50965ad P adfc52788f744448b30368cba1c1cd5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:16:35.904616 16296 raft_consensus.cc:2258] T 158d6801864e400298dcae8ad50965ad P adfc52788f744448b30368cba1c1cd5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:16:35.904929 16297 tablet_replica.cc:330] T 158d6801864e400298dcae8ad50965ad P d8b0a26a9a4141fb95452f16af8283fc: stopping tablet replica
I20260924 04:16:35.905210 16297 raft_consensus.cc:2229] T 158d6801864e400298dcae8ad50965ad P d8b0a26a9a4141fb95452f16af8283fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:16:35.905498 16297 raft_consensus.cc:2258] T 158d6801864e400298dcae8ad50965ad P d8b0a26a9a4141fb95452f16af8283fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:16:35.906275 16296 ts_tablet_manager.cc:1905] T 158d6801864e400298dcae8ad50965ad P adfc52788f744448b30368cba1c1cd5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:16:35.906903 16297 ts_tablet_manager.cc:1905] T 158d6801864e400298dcae8ad50965ad P d8b0a26a9a4141fb95452f16af8283fc: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 04:16:35.908923 15782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37857 (error 108)
I20260924 04:16:35.908953 16296 ts_tablet_manager.cc:1918] T 158d6801864e400298dcae8ad50965ad P adfc52788f744448b30368cba1c1cd5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:16:35.909037 16296 log.cc:1196] T 158d6801864e400298dcae8ad50965ad P adfc52788f744448b30368cba1c1cd5e: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-1/wal/wals/158d6801864e400298dcae8ad50965ad
I20260924 04:16:35.909173 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 15992
I20260924 04:16:35.910449 16267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:35.910187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37857 (local address 127.0.0.1:48370)
0924 04:16:35.910301 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:35.910368 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 04:16:35.910652 15782 catalog_manager.cc:4584] TS 7a632792b0b5464bb290c8420560401d (127.0.98.1:37857): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 158d6801864e400298dcae8ad50965ad: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37857: connect: Connection refused (error 111)
I20260924 04:16:35.910929 16297 ts_tablet_manager.cc:1918] T 158d6801864e400298dcae8ad50965ad P d8b0a26a9a4141fb95452f16af8283fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:16:35.911026 16297 log.cc:1196] T 158d6801864e400298dcae8ad50965ad P d8b0a26a9a4141fb95452f16af8283fc: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-2/wal/wals/158d6801864e400298dcae8ad50965ad
I20260924 04:16:35.911381 16297 ts_tablet_manager.cc:1939] T 158d6801864e400298dcae8ad50965ad P d8b0a26a9a4141fb95452f16af8283fc: Deleting consensus metadata
I20260924 04:16:35.912667 15781 catalog_manager.cc:4841] TS d8b0a26a9a4141fb95452f16af8283fc (127.0.98.3:37753): tablet 158d6801864e400298dcae8ad50965ad (table test-table [id=fb16417e3901416ca2309c0b5308b7d6]) successfully deleted
I20260924 04:16:35.914690 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 16126
I20260924 04:16:35.915112 16267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:35.914814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44431 (local address 127.0.0.1:36366)
0924 04:16:35.914999 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:35.915042 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:44431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260924 04:16:35.919056 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 15764
I20260924 04:16:55.924418 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42685
--webserver_interface=127.0.98.62
--webserver_port=41215
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42685
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:16:56.029948 16299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:16:56.030210 16299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:16:56.030248 16299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:16:56.034281 16299 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 04:16:56.034333 16299 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 04:16:56.034358 16299 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 04:16:56.034370 16299 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:16:56.034386 16299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:16:56.034404 16299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:16:56.034430 16299 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:16:56.034458 16299 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:16:56.039629 16299 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42685
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=41215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:16:56.040618 16299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:16:56.041606 16299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:16:56.054091 16299 server_base.cc:1034] running on GCE node
W20260924 04:16:56.054036 16305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:16:56.054062 16306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:16:56.054040 16308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:16:56.054876 16299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:16:56.056010 16299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:16:56.057152 16299 hybrid_clock.cc:648] HybridClock initialized: now 1790223416057125 us; error 40 us; skew 500 ppm
Sep 24 04:16:56 dist-test-slave-p401 krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:16:56.059726 16299 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:16:56.060585 16299 webserver.cc:460] Webserver started at http://127.0.98.62:41215/ using document root <none> and password file <none>
I20260924 04:16:56.061131 16299 fs_manager.cc:359] Metadata directory not provided
I20260924 04:16:56.061203 16299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:16:56.064846 16299 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:16:56.066776 16315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:16:56.067932 16299 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:16:56.068050 16299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/master-0/wal
uuid: "a03a0021d8a04239b1acb304a66797ac"
format_stamp: "Formatted at 2026-09-24 04:16:35 on dist-test-slave-p401"
server_key: "7118c983872934d3829802ecad811fa3"
server_key_iv: "2fd615e22e241d5dfaedc89b258fdeb9"
server_key_version: "encryptionkey@0"
I20260924 04:16:56.068596 16299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:16:56.093497 16299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:16:56.096189 16299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:16:56.096359 16299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:16:56.102952 16299 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42685
I20260924 04:16:56.103201 16367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42685 every 8 connection(s)
I20260924 04:16:56.104100 16299 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:16:56.108268 16368 sys_catalog.cc:263] Verifying existing consensus state
I20260924 04:16:56.109843 16368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac: Bootstrap starting.
I20260924 04:16:56.110395 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 16299
I20260924 04:16:56.110852 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37857
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:42685
--webserver_port=44351
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:16:56.126875 16368 log.cc:824] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac: Log is configured to *not* fsync() on all Append() calls
I20260924 04:16:56.135336 16368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:16:56.135654 16368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac: Bootstrap complete.
I20260924 04:16:56.139468 16368 raft_consensus.cc:348] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a03a0021d8a04239b1acb304a66797ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42685 } }
I20260924 04:16:56.139848 16368 raft_consensus.cc:729] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a03a0021d8a04239b1acb304a66797ac, State: Initialized, Role: FOLLOWER
I20260924 04:16:56.140317 16368 consensus_queue.cc:260] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a03a0021d8a04239b1acb304a66797ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42685 } }
I20260924 04:16:56.140430 16368 raft_consensus.cc:388] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:16:56.140494 16368 raft_consensus.cc:482] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:16:56.140585 16368 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac [term 1 FOLLOWER]: Advancing to term 2
I20260924 04:16:56.142361 16368 raft_consensus.cc:504] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a03a0021d8a04239b1acb304a66797ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42685 } }
I20260924 04:16:56.142657 16368 leader_election.cc:302] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a03a0021d8a04239b1acb304a66797ac; no voters:
I20260924 04:16:56.142920 16368 leader_election.cc:288] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 04:16:56.143045 16372 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac [term 2 FOLLOWER]: Leader election won for term 2
I20260924 04:16:56.143296 16372 raft_consensus.cc:686] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac [term 2 LEADER]: Becoming Leader. State: Replica: a03a0021d8a04239b1acb304a66797ac, State: Running, Role: LEADER
I20260924 04:16:56.143654 16372 consensus_queue.cc:237] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a03a0021d8a04239b1acb304a66797ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42685 } }
I20260924 04:16:56.144047 16368 sys_catalog.cc:564] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:16:56.144912 16374 sys_catalog.cc:455] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader a03a0021d8a04239b1acb304a66797ac. Latest consensus state: current_term: 2 leader_uuid: "a03a0021d8a04239b1acb304a66797ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a03a0021d8a04239b1acb304a66797ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42685 } } }
I20260924 04:16:56.145021 16374 sys_catalog.cc:458] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac [sys.catalog]: This master's current role is: LEADER
I20260924 04:16:56.145292 16373 sys_catalog.cc:455] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a03a0021d8a04239b1acb304a66797ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a03a0021d8a04239b1acb304a66797ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42685 } } }
I20260924 04:16:56.145355 16373 sys_catalog.cc:458] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac [sys.catalog]: This master's current role is: LEADER
I20260924 04:16:56.148635 16388 catalog_manager.cc:1259] Loaded cluster ID: 7e4a25d231f7467a98226bbbb4df0995
I20260924 04:16:56.148690 16388 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac: loading cluster ID for follower catalog manager: success
I20260924 04:16:56.151113 16388 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac: acquiring CA information for follower catalog manager: success
I20260924 04:16:56.152649 16388 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 04:16:56.152772 16381 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:16:56.155112 16381 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:16:56.155632 16381 catalog_manager.cc:1259] Loaded cluster ID: 7e4a25d231f7467a98226bbbb4df0995
I20260924 04:16:56.155691 16381 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:16:56.156333 16381 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:16:56.156818 16381 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac: Loaded TSK: 0
I20260924 04:16:56.169734 16381 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac: Generated new TSK 1
I20260924 04:16:56.169986 16381 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 04:16:56.220511 16370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:16:56.220777 16370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:16:56.220834 16370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:16:56.224841 16370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:16:56.224897 16370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:16:56.224979 16370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:16:56.230123 16370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37857
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=44351
--tserver_master_addrs=127.0.98.62:42685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:16:56.231093 16370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:16:56.232110 16370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:16:56.241199 16398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:16:56.241223 16395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:16:56.241223 16396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:16:56.241801 16370 server_base.cc:1034] running on GCE node
I20260924 04:16:56.242203 16370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:16:56.242815 16370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:16:56.243979 16370 hybrid_clock.cc:648] HybridClock initialized: now 1790223416243963 us; error 26 us; skew 500 ppm
Sep 24 04:16:56 dist-test-slave-p401 krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:16:56.246762 16370 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:16:56.247669 16370 webserver.cc:460] Webserver started at http://127.0.98.1:44351/ using document root <none> and password file <none>
I20260924 04:16:56.248185 16370 fs_manager.cc:359] Metadata directory not provided
I20260924 04:16:56.248253 16370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:16:56.251885 16370 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:16:56.253980 16405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:16:56.255309 16370 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:16:56.255412 16370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-0/wal
uuid: "7a632792b0b5464bb290c8420560401d"
format_stamp: "Formatted at 2026-09-24 04:16:35 on dist-test-slave-p401"
server_key: "e6425b62388f47ff1a883d17ffe9191d"
server_key_iv: "dc187a9f40f73ced6e2a5aed30e12197"
server_key_version: "encryptionkey@0"
I20260924 04:16:56.255926 16370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:16:56.269351 16370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:16:56.272306 16370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:16:56.272462 16370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:16:56.273016 16370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:16:56.274104 16412 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 04:16:56.279386 16370 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 04:16:56.279441 16370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260924 04:16:56.279503 16370 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 04:16:56.281090 16370 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 04:16:56.281132 16370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260924 04:16:56.281433 16412 tablet_bootstrap.cc:492] T 158d6801864e400298dcae8ad50965ad P 7a632792b0b5464bb290c8420560401d: Bootstrap starting.
I20260924 04:16:56.290364 16370 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37857
I20260924 04:16:56.290601 16519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37857 every 8 connection(s)
I20260924 04:16:56.291347 16370 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:16:56.297096 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 16370
I20260924 04:16:56.297580 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:44431
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:42685
--webserver_port=45289
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:16:56.301013 16412 log.cc:824] T 158d6801864e400298dcae8ad50965ad P 7a632792b0b5464bb290c8420560401d: Log is configured to *not* fsync() on all Append() calls
Sep 24 04:16:56 dist-test-slave-p401 krb5kdc[15751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:16:56.307698 16522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:56.293411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.1:48619)
0924 04:16:56.293866 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:56.293915 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 04:16:56.294703 (+ 788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:16:56.296041 (+ 1338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:16:56.296047 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:16:56.296371 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:16:56.296841 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:56.296855 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:56.298219 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:56.298222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:56.299075 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:56.299085 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:56.299791 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:56.299795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:56.299984 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:56.300507 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:16:56.300524 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:16:56.301991 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:16:56.304413 (+ 2422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:56.304419 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:56.304432 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:56.304745 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:56.305125 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:56.305128 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:56.305130 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:56.305285 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:56.305961 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:16:56.305967 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:16:56.306261 (+ 294us) client_negotiation.cc:769] Sending connection context
0924 04:16:56.306451 (+ 190us) client_negotiation.cc:240] Negotiation successful
0924 04:16:56.306761 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":109,"threads_started":1}
I20260924 04:16:56.307793 16523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:56.293472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48619 (local address 127.0.98.62:42685)
0924 04:16:56.293917 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:56.293934 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:56.294016 (+ 82us) server_negotiation.cc:407] Connection header received
0924 04:16:56.295138 (+ 1122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:56.295153 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:56.295588 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:56.296042 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:16:56.297002 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:56.298078 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:56.299187 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:56.299676 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:56.299820 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:56.302194 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:16:56.302200 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:16:56.302218 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:16:56.302249 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:16:56.304304 (+ 2055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:56.304860 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:56.304866 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:56.304872 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:56.305020 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:56.305374 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:56.305378 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:56.305380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:56.305828 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:16:56.305955 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:56.306600 (+ 645us) server_negotiation.cc:299] Negotiation successful
0924 04:16:56.306898 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":114,"threads_started":1}
I20260924 04:16:56.309376 16412 tablet_bootstrap.cc:492] T 158d6801864e400298dcae8ad50965ad P 7a632792b0b5464bb290c8420560401d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:16:56.309707 16412 tablet_bootstrap.cc:492] T 158d6801864e400298dcae8ad50965ad P 7a632792b0b5464bb290c8420560401d: Bootstrap complete.
I20260924 04:16:56.310329 16520 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42685
I20260924 04:16:56.310568 16520 heartbeater.cc:464] Registering TS with master...
I20260924 04:16:56.310815 16412 ts_tablet_manager.cc:1397] T 158d6801864e400298dcae8ad50965ad P 7a632792b0b5464bb290c8420560401d: Time spent bootstrapping tablet: real 0.030s user 0.026s sys 0.000s
I20260924 04:16:56.311074 16520 heartbeater.cc:511] Master 127.0.98.62:42685 requested a full tablet report, sending...
I20260924 04:16:56.313248 16412 raft_consensus.cc:348] T 158d6801864e400298dcae8ad50965ad P 7a632792b0b5464bb290c8420560401d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b0a26a9a4141fb95452f16af8283fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } } peers { permanent_uuid: "7a632792b0b5464bb290c8420560401d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37857 } } peers { permanent_uuid: "adfc52788f744448b30368cba1c1cd5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44431 } }
I20260924 04:16:56.313745 16412 raft_consensus.cc:729] T 158d6801864e400298dcae8ad50965ad P 7a632792b0b5464bb290c8420560401d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a632792b0b5464bb290c8420560401d, State: Initialized, Role: FOLLOWER
I20260924 04:16:56.313727 16332 ts_manager.cc:194] Registered new tserver with Master: 7a632792b0b5464bb290c8420560401d (127.0.98.1:37857)
I20260924 04:16:56.314198 16412 consensus_queue.cc:260] T 158d6801864e400298dcae8ad50965ad P 7a632792b0b5464bb290c8420560401d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b0a26a9a4141fb95452f16af8283fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } } peers { permanent_uuid: "7a632792b0b5464bb290c8420560401d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37857 } } peers { permanent_uuid: "adfc52788f744448b30368cba1c1cd5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44431 } }
I20260924 04:16:56.315174 16412 ts_tablet_manager.cc:1428] T 158d6801864e400298dcae8ad50965ad P 7a632792b0b5464bb290c8420560401d: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20260924 04:16:56.317196 16332 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48619
I20260924 04:16:56.319674 16520 heartbeater.cc:503] Master 127.0.98.62:42685 was elected leader, sending a full tablet report...
Sep 24 04:16:56 dist-test-slave-p401 krb5kdc[15751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:16:56.325819 16528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:56.316719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43378 (local address 127.0.98.1:37857)
0924 04:16:56.317048 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:56.317058 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:56.317410 (+ 352us) server_negotiation.cc:407] Connection header received
0924 04:16:56.317571 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:56.317576 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:56.317758 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:56.317883 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:16:56.318477 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:56.319183 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:56.320048 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:56.320527 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:56.320626 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:56.322803 (+ 2177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:16:56.322810 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:16:56.322819 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:16:56.322844 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:16:56.324188 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:56.324669 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:56.324672 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:56.324674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:56.324742 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:56.324979 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:56.324982 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:56.324983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:56.325176 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:16:56.325266 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:56.325588 (+ 322us) server_negotiation.cc:299] Negotiation successful
0924 04:16:56.325705 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":94,"threads_started":1}
I20260924 04:16:56.325853 16527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:56.316623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37857 (local address 127.0.0.1:43378)
0924 04:16:56.317282 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:56.317322 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 04:16:56.317445 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:16:56.317885 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:16:56.317890 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:16:56.318269 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:16:56.318366 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:56.318372 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:56.319302 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:56.319305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:56.319932 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:56.319943 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:56.320638 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:56.320653 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:56.320753 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:56.321258 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:16:56.321269 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:16:56.322670 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:16:56.324288 (+ 1618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:56.324296 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:56.324301 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:56.324579 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:56.324813 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:56.324816 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:56.324818 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:56.324907 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:56.325269 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:16:56.325274 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:16:56.325387 (+ 113us) client_negotiation.cc:769] Sending connection context
0924 04:16:56.325533 (+ 146us) client_negotiation.cc:240] Negotiation successful
0924 04:16:56.325701 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":86,"threads_started":1}
I20260924 04:16:56.326742 16454 tablet_service.cc:1514] Processing DeleteTablet for tablet 158d6801864e400298dcae8ad50965ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:16:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43378
I20260924 04:16:56.327622 16529 tablet_replica.cc:330] stopping tablet replica
I20260924 04:16:56.327883 16529 raft_consensus.cc:2229] T 158d6801864e400298dcae8ad50965ad P 7a632792b0b5464bb290c8420560401d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:16:56.328091 16529 raft_consensus.cc:2258] T 158d6801864e400298dcae8ad50965ad P 7a632792b0b5464bb290c8420560401d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:16:56.328913 16529 ts_tablet_manager.cc:1905] T 158d6801864e400298dcae8ad50965ad P 7a632792b0b5464bb290c8420560401d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:16:56.332267 16529 ts_tablet_manager.cc:1918] T 158d6801864e400298dcae8ad50965ad P 7a632792b0b5464bb290c8420560401d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:16:56.332348 16529 log.cc:1196] T 158d6801864e400298dcae8ad50965ad P 7a632792b0b5464bb290c8420560401d: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-0/wal/wals/158d6801864e400298dcae8ad50965ad
I20260924 04:16:56.332554 16529 ts_tablet_manager.cc:1939] T 158d6801864e400298dcae8ad50965ad P 7a632792b0b5464bb290c8420560401d: Deleting consensus metadata
I20260924 04:16:56.334144 16318 catalog_manager.cc:4841] TS 7a632792b0b5464bb290c8420560401d (127.0.98.1:37857): tablet 158d6801864e400298dcae8ad50965ad (table test-table [id=fb16417e3901416ca2309c0b5308b7d6]) successfully deleted
W20260924 04:16:56.410063 16524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:16:56.410322 16524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:16:56.410364 16524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:16:56.414304 16524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:16:56.414361 16524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:16:56.414422 16524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:16:56.419467 16524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:44431
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=45289
--tserver_master_addrs=127.0.98.62:42685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:16:56.420408 16524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:16:56.421334 16524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:16:56.430127 16537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:16:56.430218 16534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:16:56.430135 16535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:16:56.430374 16524 server_base.cc:1034] running on GCE node
I20260924 04:16:56.430828 16524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:16:56.431442 16524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:16:56.432600 16524 hybrid_clock.cc:648] HybridClock initialized: now 1790223416432582 us; error 30 us; skew 500 ppm
Sep 24 04:16:56 dist-test-slave-p401 krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:16:56.435061 16524 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:16:56.435905 16524 webserver.cc:460] Webserver started at http://127.0.98.2:45289/ using document root <none> and password file <none>
I20260924 04:16:56.436406 16524 fs_manager.cc:359] Metadata directory not provided
I20260924 04:16:56.436493 16524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:16:56.440037 16524 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260924 04:16:56.441933 16544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:16:56.443054 16524 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:16:56.443163 16524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-1/wal
uuid: "adfc52788f744448b30368cba1c1cd5e"
format_stamp: "Formatted at 2026-09-24 04:16:35 on dist-test-slave-p401"
server_key: "7c2671fb1614616197695f60d2825b1e"
server_key_iv: "9ee2272ba04d60a708429de355b63914"
server_key_version: "encryptionkey@0"
I20260924 04:16:56.443725 16524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:16:56.453605 16524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:16:56.456158 16524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:16:56.456312 16524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:16:56.456833 16524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:16:56.457767 16551 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260924 04:16:56.461403 16551 ts_tablet_manager.cc:1852] T 158d6801864e400298dcae8ad50965ad P adfc52788f744448b30368cba1c1cd5e: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260924 04:16:56.461772 16551 ts_tablet_manager.cc:1905] T 158d6801864e400298dcae8ad50965ad P adfc52788f744448b30368cba1c1cd5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:16:56.465551 16551 ts_tablet_manager.cc:1918] T 158d6801864e400298dcae8ad50965ad P adfc52788f744448b30368cba1c1cd5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:16:56.465643 16551 log.cc:1196] T 158d6801864e400298dcae8ad50965ad P adfc52788f744448b30368cba1c1cd5e: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-1/wal/wals/158d6801864e400298dcae8ad50965ad
I20260924 04:16:56.465845 16551 ts_tablet_manager.cc:1939] T 158d6801864e400298dcae8ad50965ad P adfc52788f744448b30368cba1c1cd5e: Deleting consensus metadata
I20260924 04:16:56.466140 16524 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260924 04:16:56.466194 16524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.000s
I20260924 04:16:56.466262 16524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:16:56.466287 16524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:16:56.474454 16524 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44431
I20260924 04:16:56.474507 16658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44431 every 8 connection(s)
I20260924 04:16:56.475265 16524 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 04:16:56 dist-test-slave-p401 krb5kdc[15751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:16:56.485116 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 16524
I20260924 04:16:56.485571 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37753
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:42685
--webserver_port=40435
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:16:56.487586 16523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:56.476903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50661 (local address 127.0.98.62:42685)
0924 04:16:56.477028 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:56.477032 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:56.477819 (+ 787us) server_negotiation.cc:407] Connection header received
0924 04:16:56.478702 (+ 883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:56.478705 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:56.478760 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:56.478832 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:16:56.479968 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:56.480501 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:56.481570 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:56.481800 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:56.481887 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:56.484154 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:16:56.484164 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:16:56.484168 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:16:56.484196 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:16:56.485523 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:56.486069 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:56.486072 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:56.486074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:56.486151 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:56.486451 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:56.486453 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:56.486455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:56.486628 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:16:56.486683 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:56.487294 (+ 611us) server_negotiation.cc:299] Negotiation successful
0924 04:16:56.487423 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 04:16:56.488245 16661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:56.477260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.2:50661)
0924 04:16:56.477684 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:56.477725 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 04:16:56.478497 (+ 772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:16:56.478989 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:16:56.478995 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:16:56.479368 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:16:56.479832 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:56.479846 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:56.480618 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:56.480621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:56.481457 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:56.481466 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:56.481930 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:56.481935 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:56.482087 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:56.482594 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:16:56.482610 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:16:56.483950 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:16:56.485642 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:56.485648 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:56.485662 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:56.485946 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:56.486237 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:56.486239 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:56.486241 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:56.486374 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:56.486750 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:16:56.486756 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:16:56.487042 (+ 286us) client_negotiation.cc:769] Sending connection context
0924 04:16:56.487194 (+ 152us) client_negotiation.cc:240] Negotiation successful
0924 04:16:56.487459 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":100,"threads_started":1}
I20260924 04:16:56.489269 16659 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42685
I20260924 04:16:56.489482 16659 heartbeater.cc:464] Registering TS with master...
I20260924 04:16:56.489905 16659 heartbeater.cc:511] Master 127.0.98.62:42685 requested a full tablet report, sending...
I20260924 04:16:56.490818 16332 ts_manager.cc:194] Registered new tserver with Master: adfc52788f744448b30368cba1c1cd5e (127.0.98.2:44431)
I20260924 04:16:56.491385 16332 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50661
W20260924 04:16:56.591112 16662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:16:56.591389 16662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:16:56.591450 16662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:16:56.595489 16662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:16:56.595549 16662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:16:56.595633 16662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:16:56.600837 16662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37753
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40435
--tserver_master_addrs=127.0.98.62:42685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:16:56.601837 16662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:16:56.602782 16662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:16:56.611778 16670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:16:56.611840 16668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:16:56.611951 16662 server_base.cc:1034] running on GCE node
W20260924 04:16:56.611779 16667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:16:56.612474 16662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:16:56.613054 16662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:16:56.614212 16662 hybrid_clock.cc:648] HybridClock initialized: now 1790223416614192 us; error 34 us; skew 500 ppm
Sep 24 04:16:56 dist-test-slave-p401 krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:16:56.616882 16662 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:16:56.617666 16662 webserver.cc:460] Webserver started at http://127.0.98.3:40435/ using document root <none> and password file <none>
I20260924 04:16:56.618176 16662 fs_manager.cc:359] Metadata directory not provided
I20260924 04:16:56.618261 16662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:16:56.621923 16662 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260924 04:16:56.623839 16677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:16:56.624964 16662 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 04:16:56.625068 16662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-2/wal
uuid: "d8b0a26a9a4141fb95452f16af8283fc"
format_stamp: "Formatted at 2026-09-24 04:16:35 on dist-test-slave-p401"
server_key: "f7ae8f9b974f75cd6d051bef2100e1d5"
server_key_iv: "a515fd50bab75fbdfdd9f322ea1a8b83"
server_key_version: "encryptionkey@0"
I20260924 04:16:56.625564 16662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:16:56.639463 16662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:16:56.641970 16662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:16:56.642132 16662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:16:56.642648 16662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:16:56.643432 16662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:16:56.643486 16662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:16:56.643537 16662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:16:56.643561 16662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:16:56.652083 16662 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37753
I20260924 04:16:56.652123 16790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37753 every 8 connection(s)
I20260924 04:16:56.653026 16662 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 04:16:56 dist-test-slave-p401 krb5kdc[15751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:16:56.661966 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 16662
I20260924 04:16:56.665832 16523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:56.654658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40119 (local address 127.0.98.62:42685)
0924 04:16:56.654787 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:56.654791 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:56.655813 (+ 1022us) server_negotiation.cc:407] Connection header received
0924 04:16:56.656628 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:56.656631 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:56.656675 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:56.656742 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:16:56.657812 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:56.658314 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:56.659396 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:56.659621 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:56.659713 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:56.661954 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:16:56.661958 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:16:56.661961 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:16:56.661997 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:16:56.663722 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:56.664270 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:56.664273 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:56.664275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:56.664377 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:56.664761 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:56.664765 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:56.664767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:56.664964 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:16:56.665028 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:56.665554 (+ 526us) server_negotiation.cc:299] Negotiation successful
0924 04:16:56.665675 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:16:56.666836 16793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:56.655042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.3:40119)
0924 04:16:56.655691 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:56.655730 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 04:16:56.656460 (+ 730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:16:56.656888 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:16:56.656895 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:16:56.657243 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:16:56.657704 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:56.657718 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:56.658394 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:56.658396 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:56.659299 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:56.659308 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:56.659750 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:56.659754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:56.659890 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:56.660398 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:16:56.660413 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:16:56.661798 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:16:56.663845 (+ 2047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:56.663851 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:56.663867 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:56.664192 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:56.664484 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:16:56.664487 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:16:56.664489 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:16:56.664658 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:16:56.665116 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:16:56.665121 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:16:56.665335 (+ 214us) client_negotiation.cc:769] Sending connection context
0924 04:16:56.665491 (+ 156us) client_negotiation.cc:240] Negotiation successful
0924 04:16:56.665796 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":150,"threads_started":1}
I20260924 04:16:56.668102 16791 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42685
I20260924 04:16:56.668336 16791 heartbeater.cc:464] Registering TS with master...
I20260924 04:16:56.668877 16791 heartbeater.cc:511] Master 127.0.98.62:42685 requested a full tablet report, sending...
I20260924 04:16:56.669937 16332 ts_manager.cc:194] Registered new tserver with Master: d8b0a26a9a4141fb95452f16af8283fc (127.0.98.3:37753)
I20260924 04:16:56.670548 16332 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40119
I20260924 04:16:56.671717 16795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:56.662640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55926 (local address 127.0.98.62:42685)
0924 04:16:56.662867 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:56.662872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:56.663157 (+ 285us) server_negotiation.cc:407] Connection header received
0924 04:16:56.663299 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:56.663301 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:56.663349 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:56.663429 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:16:56.664270 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:56.665017 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:56.666506 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:56.666834 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:56.666936 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:56.668326 (+ 1390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:16:56.668330 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:16:56.668333 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:16:56.668357 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:16:56.669904 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:56.670377 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:56.670380 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:56.670382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:56.670461 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:56.670847 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:56.670850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:56.670852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:56.671047 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:16:56.671279 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:56.671419 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 04:16:56.671534 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":78,"threads_started":1}
I20260924 04:16:56.673904 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:16:56.678392 16795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:56.674451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55942 (local address 127.0.98.62:42685)
0924 04:16:56.674548 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:56.674552 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:56.674729 (+ 177us) server_negotiation.cc:407] Connection header received
0924 04:16:56.674898 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:56.674900 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:56.674981 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:56.675060 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:16:56.675288 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:56.675980 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:56.677156 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:56.677539 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:56.677648 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:56.678008 (+ 360us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:16:56.678126 (+ 118us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0924 04:16:56.678308 (+ 182us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55942: token expired
Metrics: {"server-negotiator.queue_time_us":32}
W20260924 04:16:56.678443 16795 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55942: token expired
W20260924 04:16:56.678817 16261 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:42685)
I20260924 04:16:56.685407 16795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:56.679121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55946 (local address 127.0.98.62:42685)
0924 04:16:56.679259 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:56.679264 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:56.679276 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:16:56.679369 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:56.679372 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:56.679408 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:56.679478 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:16:56.680021 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:56.680498 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:56.681241 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:56.681451 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:56.681531 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:56.682650 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:16:56.682654 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:16:56.682656 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:16:56.682679 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:16:56.683986 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:56.684367 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:56.684370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:56.684372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:56.684450 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:16:56.684680 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:16:56.684683 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:16:56.684685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:16:56.684840 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:16:56.684948 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:56.685164 (+ 216us) server_negotiation.cc:299] Negotiation successful
0924 04:16:56.685271 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 04:16:56.687852 16332 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55946:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:16:56.690935 16332 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:16:56.697914 16454 tablet_service.cc:1467] Processing CreateTablet for tablet 115649d93b6c45f4a3db817b32b4ca62 (DEFAULT_TABLE table=test-table [id=5c292ee307ee4cc2881580031e332daa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:16:56.698551 16454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 115649d93b6c45f4a3db817b32b4ca62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:16:56.701038 16412 tablet_bootstrap.cc:492] T 115649d93b6c45f4a3db817b32b4ca62 P 7a632792b0b5464bb290c8420560401d: Bootstrap starting.
I20260924 04:16:56.701850 16412 tablet_bootstrap.cc:654] T 115649d93b6c45f4a3db817b32b4ca62 P 7a632792b0b5464bb290c8420560401d: Neither blocks nor log segments found. Creating new log.
I20260924 04:16:56.702521 16527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:56.696796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44431 (local address 127.0.0.1:44486)
0924 04:16:56.696949 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:56.696962 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:16:56.697032 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:16:56.698072 (+ 1040us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:16:56.698075 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:16:56.698092 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:16:56.698193 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:56.698200 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:56.699760 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:56.699763 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:56.700980 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:56.700990 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:56.702092 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:56.702096 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:56.702194 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:56.702243 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 04:16:56.702291 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 04:16:56.702387 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"mutex_wait_us":10}
I20260924 04:16:56.702915 16800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:56.697098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37753 (local address 127.0.0.1:47450)
0924 04:16:56.697634 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:56.697647 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:16:56.697730 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:16:56.698312 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:16:56.698315 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:16:56.698331 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:16:56.698420 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:56.698426 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:56.699630 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:56.699634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:56.700980 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:56.700990 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:56.702521 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:56.702524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:56.702617 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:56.702640 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 04:16:56.702717 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 04:16:56.702782 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"mutex_wait_us":55,"thread_start_us":68,"threads_started":1}
I20260924 04:16:56.703003 16802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:56.697363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47450 (local address 127.0.98.3:37753)
0924 04:16:56.697732 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:56.697742 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:56.697765 (+ 23us) server_negotiation.cc:407] Connection header received
0924 04:16:56.697837 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:56.697841 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:56.698005 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:56.698147 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:16:56.698525 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:56.699530 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:56.701194 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:56.702240 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:56.702331 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:56.702465 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:56.702763 (+ 298us) server_negotiation.cc:299] Negotiation successful
0924 04:16:56.702864 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":108,"threads_started":1}
I20260924 04:16:56.703215 16801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:56.697221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44486 (local address 127.0.98.2:44431)
0924 04:16:56.697559 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:56.697583 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:56.697607 (+ 24us) server_negotiation.cc:407] Connection header received
0924 04:16:56.697683 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:56.697689 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:56.697913 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:56.698086 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:16:56.698285 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:56.699660 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:56.701115 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:56.701962 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:56.702773 (+ 811us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:56.702912 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:56.702965 (+ 53us) server_negotiation.cc:299] Negotiation successful
0924 04:16:56.703052 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":129,"threads_started":1}
I20260924 04:16:56.703713 16412 tablet_bootstrap.cc:492] T 115649d93b6c45f4a3db817b32b4ca62 P 7a632792b0b5464bb290c8420560401d: No bootstrap required, opened a new log
I20260924 04:16:56.703794 16412 ts_tablet_manager.cc:1397] T 115649d93b6c45f4a3db817b32b4ca62 P 7a632792b0b5464bb290c8420560401d: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260924 04:16:56.704213 16412 raft_consensus.cc:348] T 115649d93b6c45f4a3db817b32b4ca62 P 7a632792b0b5464bb290c8420560401d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adfc52788f744448b30368cba1c1cd5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44431 } } peers { permanent_uuid: "d8b0a26a9a4141fb95452f16af8283fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } } peers { permanent_uuid: "7a632792b0b5464bb290c8420560401d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37857 } }
I20260924 04:16:56.704331 16412 raft_consensus.cc:374] T 115649d93b6c45f4a3db817b32b4ca62 P 7a632792b0b5464bb290c8420560401d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:16:56.704368 16412 raft_consensus.cc:729] T 115649d93b6c45f4a3db817b32b4ca62 P 7a632792b0b5464bb290c8420560401d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a632792b0b5464bb290c8420560401d, State: Initialized, Role: FOLLOWER
I20260924 04:16:56.704493 16412 consensus_queue.cc:260] T 115649d93b6c45f4a3db817b32b4ca62 P 7a632792b0b5464bb290c8420560401d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adfc52788f744448b30368cba1c1cd5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44431 } } peers { permanent_uuid: "d8b0a26a9a4141fb95452f16af8283fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } } peers { permanent_uuid: "7a632792b0b5464bb290c8420560401d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37857 } }
I20260924 04:16:56.704747 16412 ts_tablet_manager.cc:1428] T 115649d93b6c45f4a3db817b32b4ca62 P 7a632792b0b5464bb290c8420560401d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 04:16:56.705760 16526 raft_consensus.cc:482] T 115649d93b6c45f4a3db817b32b4ca62 P 7a632792b0b5464bb290c8420560401d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:16:56.705920 16526 raft_consensus.cc:504] T 115649d93b6c45f4a3db817b32b4ca62 P 7a632792b0b5464bb290c8420560401d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adfc52788f744448b30368cba1c1cd5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44431 } } peers { permanent_uuid: "d8b0a26a9a4141fb95452f16af8283fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } } peers { permanent_uuid: "7a632792b0b5464bb290c8420560401d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37857 } }
I20260924 04:16:56.706138 16725 tablet_service.cc:1467] Processing CreateTablet for tablet 115649d93b6c45f4a3db817b32b4ca62 (DEFAULT_TABLE table=test-table [id=5c292ee307ee4cc2881580031e332daa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:16:56.706848 16526 leader_election.cc:288] T 115649d93b6c45f4a3db817b32b4ca62 P 7a632792b0b5464bb290c8420560401d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers adfc52788f744448b30368cba1c1cd5e (127.0.98.2:44431), d8b0a26a9a4141fb95452f16af8283fc (127.0.98.3:37753)
I20260924 04:16:56.707114 16725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 115649d93b6c45f4a3db817b32b4ca62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:16:56.711383 16801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:56.707614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36113 (local address 127.0.98.2:44431)
0924 04:16:56.707808 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:56.707812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:56.707824 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:16:56.707862 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:56.707865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:56.707908 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:56.707979 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:16:56.708313 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:56.708925 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:56.710263 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:56.710822 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:56.710927 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:56.710984 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:56.711180 (+ 196us) server_negotiation.cc:299] Negotiation successful
0924 04:16:56.711272 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260924 04:16:56.711386 16522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:56.707017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44431 (local address 127.0.98.1:36113)
0924 04:16:56.707349 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:56.707362 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:16:56.707441 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:16:56.708007 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:16:56.708010 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:16:56.708029 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:16:56.708185 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:56.708193 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:56.709029 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:56.709032 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:56.710137 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:56.710147 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:56.710943 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:56.710946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:56.711041 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:56.711084 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 04:16:56.711156 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 04:16:56.711256 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261}
I20260924 04:16:56.711524 16804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:56.707067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37753 (local address 127.0.98.1:37549)
0924 04:16:56.707696 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:56.707708 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:16:56.707788 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:16:56.708361 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:16:56.708364 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:16:56.708380 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:16:56.708503 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:56.708509 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:56.709395 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:56.709398 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:56.710429 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:56.710436 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:56.711152 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:56.711155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:56.711266 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:56.711286 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 04:16:56.711345 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 04:16:56.711406 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"mutex_wait_us":2,"thread_start_us":90,"threads_started":1}
I20260924 04:16:56.711870 16802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:56.707421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37549 (local address 127.0.98.3:37753)
0924 04:16:56.708113 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:56.708117 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:56.708128 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:16:56.708164 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:56.708166 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:56.708205 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:56.708268 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:16:56.708630 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:56.709275 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:56.710532 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:56.710941 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:56.711034 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:56.711083 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:56.711695 (+ 612us) server_negotiation.cc:299] Negotiation successful
0924 04:16:56.711755 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625}
I20260924 04:16:56.711911 16593 tablet_service.cc:1467] Processing CreateTablet for tablet 115649d93b6c45f4a3db817b32b4ca62 (DEFAULT_TABLE table=test-table [id=5c292ee307ee4cc2881580031e332daa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:16:56.712802 16593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 115649d93b6c45f4a3db817b32b4ca62. 1 dirs total, 0 dirs full, 0 dirs failed
W20260924 04:16:56.713076 16409 leader_election.cc:341] T 115649d93b6c45f4a3db817b32b4ca62 P 7a632792b0b5464bb290c8420560401d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer adfc52788f744448b30368cba1c1cd5e (127.0.98.2:44431): Not found: Tablet not found: 115649d93b6c45f4a3db817b32b4ca62
W20260924 04:16:56.716215 16407 leader_election.cc:341] T 115649d93b6c45f4a3db817b32b4ca62 P 7a632792b0b5464bb290c8420560401d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8b0a26a9a4141fb95452f16af8283fc (127.0.98.3:37753): Not found: Tablet not found: 115649d93b6c45f4a3db817b32b4ca62
I20260924 04:16:56.716337 16407 leader_election.cc:302] T 115649d93b6c45f4a3db817b32b4ca62 P 7a632792b0b5464bb290c8420560401d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a632792b0b5464bb290c8420560401d; no voters: adfc52788f744448b30368cba1c1cd5e, d8b0a26a9a4141fb95452f16af8283fc
I20260924 04:16:56.716778 16526 raft_consensus.cc:2726] T 115649d93b6c45f4a3db817b32b4ca62 P 7a632792b0b5464bb290c8420560401d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 04:16:56.717231 16551 tablet_bootstrap.cc:492] T 115649d93b6c45f4a3db817b32b4ca62 P adfc52788f744448b30368cba1c1cd5e: Bootstrap starting.
I20260924 04:16:56.719010 16551 tablet_bootstrap.cc:654] T 115649d93b6c45f4a3db817b32b4ca62 P adfc52788f744448b30368cba1c1cd5e: Neither blocks nor log segments found. Creating new log.
I20260924 04:16:56.719286 16805 tablet_bootstrap.cc:492] T 115649d93b6c45f4a3db817b32b4ca62 P d8b0a26a9a4141fb95452f16af8283fc: Bootstrap starting.
I20260924 04:16:56.719890 16551 log.cc:824] T 115649d93b6c45f4a3db817b32b4ca62 P adfc52788f744448b30368cba1c1cd5e: Log is configured to *not* fsync() on all Append() calls
I20260924 04:16:56.721439 16805 tablet_bootstrap.cc:654] T 115649d93b6c45f4a3db817b32b4ca62 P d8b0a26a9a4141fb95452f16af8283fc: Neither blocks nor log segments found. Creating new log.
I20260924 04:16:56.722338 16805 log.cc:824] T 115649d93b6c45f4a3db817b32b4ca62 P d8b0a26a9a4141fb95452f16af8283fc: Log is configured to *not* fsync() on all Append() calls
I20260924 04:16:56.723724 16551 tablet_bootstrap.cc:492] T 115649d93b6c45f4a3db817b32b4ca62 P adfc52788f744448b30368cba1c1cd5e: No bootstrap required, opened a new log
I20260924 04:16:56.723902 16551 ts_tablet_manager.cc:1397] T 115649d93b6c45f4a3db817b32b4ca62 P adfc52788f744448b30368cba1c1cd5e: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.003s
I20260924 04:16:56.724416 16805 tablet_bootstrap.cc:492] T 115649d93b6c45f4a3db817b32b4ca62 P d8b0a26a9a4141fb95452f16af8283fc: No bootstrap required, opened a new log
I20260924 04:16:56.724602 16805 ts_tablet_manager.cc:1397] T 115649d93b6c45f4a3db817b32b4ca62 P d8b0a26a9a4141fb95452f16af8283fc: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:16:56.727692 16551 raft_consensus.cc:348] T 115649d93b6c45f4a3db817b32b4ca62 P adfc52788f744448b30368cba1c1cd5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adfc52788f744448b30368cba1c1cd5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44431 } } peers { permanent_uuid: "d8b0a26a9a4141fb95452f16af8283fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } } peers { permanent_uuid: "7a632792b0b5464bb290c8420560401d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37857 } }
I20260924 04:16:56.727943 16551 raft_consensus.cc:374] T 115649d93b6c45f4a3db817b32b4ca62 P adfc52788f744448b30368cba1c1cd5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:16:56.727994 16551 raft_consensus.cc:729] T 115649d93b6c45f4a3db817b32b4ca62 P adfc52788f744448b30368cba1c1cd5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adfc52788f744448b30368cba1c1cd5e, State: Initialized, Role: FOLLOWER
I20260924 04:16:56.728513 16551 consensus_queue.cc:260] T 115649d93b6c45f4a3db817b32b4ca62 P adfc52788f744448b30368cba1c1cd5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adfc52788f744448b30368cba1c1cd5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44431 } } peers { permanent_uuid: "d8b0a26a9a4141fb95452f16af8283fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } } peers { permanent_uuid: "7a632792b0b5464bb290c8420560401d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37857 } }
I20260924 04:16:56.728776 16805 raft_consensus.cc:348] T 115649d93b6c45f4a3db817b32b4ca62 P d8b0a26a9a4141fb95452f16af8283fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adfc52788f744448b30368cba1c1cd5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44431 } } peers { permanent_uuid: "d8b0a26a9a4141fb95452f16af8283fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } } peers { permanent_uuid: "7a632792b0b5464bb290c8420560401d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37857 } }
I20260924 04:16:56.729014 16805 raft_consensus.cc:374] T 115649d93b6c45f4a3db817b32b4ca62 P d8b0a26a9a4141fb95452f16af8283fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:16:56.729058 16805 raft_consensus.cc:729] T 115649d93b6c45f4a3db817b32b4ca62 P d8b0a26a9a4141fb95452f16af8283fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8b0a26a9a4141fb95452f16af8283fc, State: Initialized, Role: FOLLOWER
I20260924 04:16:56.729529 16805 consensus_queue.cc:260] T 115649d93b6c45f4a3db817b32b4ca62 P d8b0a26a9a4141fb95452f16af8283fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adfc52788f744448b30368cba1c1cd5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44431 } } peers { permanent_uuid: "d8b0a26a9a4141fb95452f16af8283fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } } peers { permanent_uuid: "7a632792b0b5464bb290c8420560401d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37857 } }
I20260924 04:16:56.729616 16551 ts_tablet_manager.cc:1428] T 115649d93b6c45f4a3db817b32b4ca62 P adfc52788f744448b30368cba1c1cd5e: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260924 04:16:56.729992 16659 heartbeater.cc:503] Master 127.0.98.62:42685 was elected leader, sending a full tablet report...
I20260924 04:16:56.731192 16805 ts_tablet_manager.cc:1428] T 115649d93b6c45f4a3db817b32b4ca62 P d8b0a26a9a4141fb95452f16af8283fc: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 04:16:56.731662 16791 heartbeater.cc:503] Master 127.0.98.62:42685 was elected leader, sending a full tablet report...
W20260924 04:16:56.792471 16521 tablet.cc:2369] T 115649d93b6c45f4a3db817b32b4ca62 P 7a632792b0b5464bb290c8420560401d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:16:56.853502 16809 raft_consensus.cc:482] T 115649d93b6c45f4a3db817b32b4ca62 P d8b0a26a9a4141fb95452f16af8283fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:16:56.853709 16809 raft_consensus.cc:504] T 115649d93b6c45f4a3db817b32b4ca62 P d8b0a26a9a4141fb95452f16af8283fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adfc52788f744448b30368cba1c1cd5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44431 } } peers { permanent_uuid: "d8b0a26a9a4141fb95452f16af8283fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } } peers { permanent_uuid: "7a632792b0b5464bb290c8420560401d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37857 } }
I20260924 04:16:56.854817 16809 leader_election.cc:288] T 115649d93b6c45f4a3db817b32b4ca62 P d8b0a26a9a4141fb95452f16af8283fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers adfc52788f744448b30368cba1c1cd5e (127.0.98.2:44431), 7a632792b0b5464bb290c8420560401d (127.0.98.1:37857)
I20260924 04:16:56.859136 16793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:56.855074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44431 (local address 127.0.98.3:36069)
0924 04:16:56.855194 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:56.855210 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:16:56.855336 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:16:56.855612 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:16:56.855615 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:16:56.855637 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:16:56.855755 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:56.855762 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:56.856709 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:56.856712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:56.857961 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:56.857972 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:56.858770 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:56.858772 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:56.858847 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:56.858866 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 04:16:56.858922 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 04:16:56.858985 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":5}
I20260924 04:16:56.859150 16801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:56.855205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36069 (local address 127.0.98.2:44431)
0924 04:16:56.855348 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:56.855354 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:56.855375 (+ 21us) server_negotiation.cc:407] Connection header received
0924 04:16:56.855433 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:56.855436 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:56.855500 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:56.855585 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:16:56.855867 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:56.856601 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:56.858090 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:56.858656 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:56.858770 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:56.858831 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:56.858957 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 04:16:56.859030 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 04:16:56.859512 16810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:56.855201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44397 (local address 127.0.98.1:37857)
0924 04:16:56.855506 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:56.855510 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:56.855846 (+ 336us) server_negotiation.cc:407] Connection header received
0924 04:16:56.855947 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:56.855950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:56.856006 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:56.856095 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:16:56.856304 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:56.856922 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:56.858253 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:56.858917 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:56.859019 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:56.859162 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:56.859318 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 04:16:56.859400 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":117,"threads_started":1}
I20260924 04:16:56.859598 16613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "115649d93b6c45f4a3db817b32b4ca62" candidate_uuid: "d8b0a26a9a4141fb95452f16af8283fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adfc52788f744448b30368cba1c1cd5e" is_pre_election: true
I20260924 04:16:56.859850 16811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:56.855074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37857 (local address 127.0.98.3:44397)
0924 04:16:56.855750 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:56.855763 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:16:56.855856 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:16:56.856100 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:16:56.856102 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:16:56.856113 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:16:56.856212 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:56.856217 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:56.857018 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:56.857022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:56.858155 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:16:56.858162 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:16:56.859059 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:16:56.859062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:16:56.859148 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:56.859171 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 04:16:56.859639 (+ 468us) client_negotiation.cc:240] Negotiation successful
0924 04:16:56.859717 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"mutex_wait_us":347,"thread_start_us":135,"threads_started":1}
I20260924 04:16:56.859929 16613 raft_consensus.cc:2454] T 115649d93b6c45f4a3db817b32b4ca62 P adfc52788f744448b30368cba1c1cd5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8b0a26a9a4141fb95452f16af8283fc in term 0.
I20260924 04:16:56.860340 16474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "115649d93b6c45f4a3db817b32b4ca62" candidate_uuid: "d8b0a26a9a4141fb95452f16af8283fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a632792b0b5464bb290c8420560401d" is_pre_election: true
I20260924 04:16:56.860553 16474 raft_consensus.cc:2454] T 115649d93b6c45f4a3db817b32b4ca62 P 7a632792b0b5464bb290c8420560401d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8b0a26a9a4141fb95452f16af8283fc in term 0.
I20260924 04:16:56.860877 16681 leader_election.cc:302] T 115649d93b6c45f4a3db817b32b4ca62 P d8b0a26a9a4141fb95452f16af8283fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7a632792b0b5464bb290c8420560401d, adfc52788f744448b30368cba1c1cd5e, d8b0a26a9a4141fb95452f16af8283fc; no voters:
I20260924 04:16:56.861126 16809 raft_consensus.cc:2781] T 115649d93b6c45f4a3db817b32b4ca62 P d8b0a26a9a4141fb95452f16af8283fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:16:56.861198 16809 raft_consensus.cc:482] T 115649d93b6c45f4a3db817b32b4ca62 P d8b0a26a9a4141fb95452f16af8283fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:16:56.861227 16809 raft_consensus.cc:3037] T 115649d93b6c45f4a3db817b32b4ca62 P d8b0a26a9a4141fb95452f16af8283fc [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:16:56.862215 16809 raft_consensus.cc:504] T 115649d93b6c45f4a3db817b32b4ca62 P d8b0a26a9a4141fb95452f16af8283fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adfc52788f744448b30368cba1c1cd5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44431 } } peers { permanent_uuid: "d8b0a26a9a4141fb95452f16af8283fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } } peers { permanent_uuid: "7a632792b0b5464bb290c8420560401d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37857 } }
I20260924 04:16:56.862514 16809 leader_election.cc:288] T 115649d93b6c45f4a3db817b32b4ca62 P d8b0a26a9a4141fb95452f16af8283fc [CANDIDATE]: Term 1 election: Requested vote from peers adfc52788f744448b30368cba1c1cd5e (127.0.98.2:44431), 7a632792b0b5464bb290c8420560401d (127.0.98.1:37857)
I20260924 04:16:56.862814 16613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "115649d93b6c45f4a3db817b32b4ca62" candidate_uuid: "d8b0a26a9a4141fb95452f16af8283fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adfc52788f744448b30368cba1c1cd5e"
I20260924 04:16:56.862814 16474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "115649d93b6c45f4a3db817b32b4ca62" candidate_uuid: "d8b0a26a9a4141fb95452f16af8283fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a632792b0b5464bb290c8420560401d"
I20260924 04:16:56.862931 16474 raft_consensus.cc:3037] T 115649d93b6c45f4a3db817b32b4ca62 P 7a632792b0b5464bb290c8420560401d [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:16:56.862932 16613 raft_consensus.cc:3037] T 115649d93b6c45f4a3db817b32b4ca62 P adfc52788f744448b30368cba1c1cd5e [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:16:56.864107 16474 raft_consensus.cc:2454] T 115649d93b6c45f4a3db817b32b4ca62 P 7a632792b0b5464bb290c8420560401d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8b0a26a9a4141fb95452f16af8283fc in term 1.
I20260924 04:16:56.864107 16613 raft_consensus.cc:2454] T 115649d93b6c45f4a3db817b32b4ca62 P adfc52788f744448b30368cba1c1cd5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8b0a26a9a4141fb95452f16af8283fc in term 1.
I20260924 04:16:56.864382 16681 leader_election.cc:302] T 115649d93b6c45f4a3db817b32b4ca62 P d8b0a26a9a4141fb95452f16af8283fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: adfc52788f744448b30368cba1c1cd5e, d8b0a26a9a4141fb95452f16af8283fc; no voters:
I20260924 04:16:56.864567 16809 raft_consensus.cc:2781] T 115649d93b6c45f4a3db817b32b4ca62 P d8b0a26a9a4141fb95452f16af8283fc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:16:56.864761 16809 raft_consensus.cc:686] T 115649d93b6c45f4a3db817b32b4ca62 P d8b0a26a9a4141fb95452f16af8283fc [term 1 LEADER]: Becoming Leader. State: Replica: d8b0a26a9a4141fb95452f16af8283fc, State: Running, Role: LEADER
I20260924 04:16:56.865126 16809 consensus_queue.cc:237] T 115649d93b6c45f4a3db817b32b4ca62 P d8b0a26a9a4141fb95452f16af8283fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adfc52788f744448b30368cba1c1cd5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44431 } } peers { permanent_uuid: "d8b0a26a9a4141fb95452f16af8283fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } } peers { permanent_uuid: "7a632792b0b5464bb290c8420560401d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37857 } }
I20260924 04:16:56.867980 16331 catalog_manager.cc:5495] T 115649d93b6c45f4a3db817b32b4ca62 P d8b0a26a9a4141fb95452f16af8283fc reported cstate change: term changed from 0 to 1, leader changed from <none> to d8b0a26a9a4141fb95452f16af8283fc (127.0.98.3). New cstate: current_term: 1 leader_uuid: "d8b0a26a9a4141fb95452f16af8283fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adfc52788f744448b30368cba1c1cd5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8b0a26a9a4141fb95452f16af8283fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a632792b0b5464bb290c8420560401d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37857 } health_report { overall_health: UNKNOWN } } }
I20260924 04:16:56.901105 16802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:56.898075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47462 (local address 127.0.98.3:37753)
0924 04:16:56.898202 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 04:16:56.898205 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:16:56.898219 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:16:56.898292 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:16:56.898294 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:16:56.898344 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:16:56.898437 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:16:56.898638 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:56.899128 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:56.899888 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:16:56.900087 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:16:56.900154 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:16:56.900374 (+ 220us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:16:56.900796 (+ 422us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:16:56.900882 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 04:16:56.900926 (+ 44us) server_negotiation.cc:299] Negotiation successful
0924 04:16:56.900976 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260924 04:16:56.903992 16792 tablet.cc:2369] T 115649d93b6c45f4a3db817b32b4ca62 P d8b0a26a9a4141fb95452f16af8283fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:16:56.906713 16613 raft_consensus.cc:1261] T 115649d93b6c45f4a3db817b32b4ca62 P adfc52788f744448b30368cba1c1cd5e [term 1 FOLLOWER]: Refusing update from remote peer d8b0a26a9a4141fb95452f16af8283fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:16:56.906737 16474 raft_consensus.cc:1261] T 115649d93b6c45f4a3db817b32b4ca62 P 7a632792b0b5464bb290c8420560401d [term 1 FOLLOWER]: Refusing update from remote peer d8b0a26a9a4141fb95452f16af8283fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:16:56.907357 16812 consensus_queue.cc:1035] T 115649d93b6c45f4a3db817b32b4ca62 P d8b0a26a9a4141fb95452f16af8283fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "adfc52788f744448b30368cba1c1cd5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:16:56.907526 16809 consensus_queue.cc:1035] T 115649d93b6c45f4a3db817b32b4ca62 P d8b0a26a9a4141fb95452f16af8283fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a632792b0b5464bb290c8420560401d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:16:56.922134 16332 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:16:56.922317 16332 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:16:56.924752 16332 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac: Sending DeleteTablet for 3 replicas of tablet 115649d93b6c45f4a3db817b32b4ca62
I20260924 04:16:56.925400 16454 tablet_service.cc:1514] Processing DeleteTablet for tablet 115649d93b6c45f4a3db817b32b4ca62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:16:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43378
I20260924 04:16:56.925442 16725 tablet_service.cc:1514] Processing DeleteTablet for tablet 115649d93b6c45f4a3db817b32b4ca62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:16:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47450
I20260924 04:16:56.925436 16593 tablet_service.cc:1514] Processing DeleteTablet for tablet 115649d93b6c45f4a3db817b32b4ca62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:16:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44486
I20260924 04:16:56.925837 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 16370
I20260924 04:16:56.927016 16826 tablet_replica.cc:330] T 115649d93b6c45f4a3db817b32b4ca62 P d8b0a26a9a4141fb95452f16af8283fc: stopping tablet replica
I20260924 04:16:56.927345 16826 raft_consensus.cc:2229] T 115649d93b6c45f4a3db817b32b4ca62 P d8b0a26a9a4141fb95452f16af8283fc [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:16:56.927743 16826 raft_consensus.cc:2258] T 115649d93b6c45f4a3db817b32b4ca62 P d8b0a26a9a4141fb95452f16af8283fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:16:56.929015 16826 ts_tablet_manager.cc:1905] T 115649d93b6c45f4a3db817b32b4ca62 P d8b0a26a9a4141fb95452f16af8283fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:16:56.931353 16826 ts_tablet_manager.cc:1918] T 115649d93b6c45f4a3db817b32b4ca62 P d8b0a26a9a4141fb95452f16af8283fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:16:56.931439 16826 log.cc:1196] T 115649d93b6c45f4a3db817b32b4ca62 P d8b0a26a9a4141fb95452f16af8283fc: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-2/wal/wals/115649d93b6c45f4a3db817b32b4ca62
I20260924 04:16:56.931732 16826 ts_tablet_manager.cc:1939] T 115649d93b6c45f4a3db817b32b4ca62 P d8b0a26a9a4141fb95452f16af8283fc: Deleting consensus metadata
I20260924 04:16:56.932801 16317 catalog_manager.cc:4841] TS d8b0a26a9a4141fb95452f16af8283fc (127.0.98.3:37753): tablet 115649d93b6c45f4a3db817b32b4ca62 (table test-table [id=5c292ee307ee4cc2881580031e332daa]) successfully deleted
W20260924 04:16:56.933414 16318 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37857 (error 108)
I20260924 04:16:56.934401 16827 tablet_replica.cc:330] T 115649d93b6c45f4a3db817b32b4ca62 P adfc52788f744448b30368cba1c1cd5e: stopping tablet replica
I20260924 04:16:56.934451 16800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:56.934185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37857 (local address 127.0.0.1:43380)
0924 04:16:56.934302 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:56.934361 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 04:16:56.934535 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 16524
W20260924 04:16:56.934583 16318 catalog_manager.cc:4584] TS 7a632792b0b5464bb290c8420560401d (127.0.98.1:37857): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 115649d93b6c45f4a3db817b32b4ca62: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37857: connect: Connection refused (error 111)
I20260924 04:16:56.939826 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 16662
I20260924 04:16:56.940728 16800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:16:56.940038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44431 (local address 127.0.0.1:44488)
0924 04:16:56.940587 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0924 04:16:56.940644 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:44431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470}
I20260924 04:16:56.944553 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 16299
I20260924 04:17:16.951887 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42685
--webserver_interface=127.0.98.62
--webserver_port=41215
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42685
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:17:17.058477 16829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:17:17.058745 16829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:17:17.058785 16829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:17:17.062748 16829 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 04:17:17.062805 16829 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 04:17:17.062819 16829 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 04:17:17.062831 16829 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:17:17.062849 16829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:17:17.062868 16829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:17:17.062897 16829 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:17:17.062924 16829 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:17:17.068171 16829 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42685
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=41215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:17:17.069249 16829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:17:17.070242 16829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:17:17.078748 16837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:17:17.078881 16835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:17:17.078871 16834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:17:17.078860 16829 server_base.cc:1034] running on GCE node
I20260924 04:17:17.079728 16829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:17:17.080840 16829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:17:17.081993 16829 hybrid_clock.cc:648] HybridClock initialized: now 1790223437081975 us; error 31 us; skew 500 ppm
Sep 24 04:17:17 dist-test-slave-p401 krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:17:17.084672 16829 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:17:17.085513 16829 webserver.cc:460] Webserver started at http://127.0.98.62:41215/ using document root <none> and password file <none>
I20260924 04:17:17.086024 16829 fs_manager.cc:359] Metadata directory not provided
I20260924 04:17:17.086083 16829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:17:17.089831 16829 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:17:17.091951 16844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:17.093145 16829 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 04:17:17.093263 16829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/master-0/wal
uuid: "a03a0021d8a04239b1acb304a66797ac"
format_stamp: "Formatted at 2026-09-24 04:16:35 on dist-test-slave-p401"
server_key: "7118c983872934d3829802ecad811fa3"
server_key_iv: "2fd615e22e241d5dfaedc89b258fdeb9"
server_key_version: "encryptionkey@0"
I20260924 04:17:17.093783 16829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:17:17.113420 16829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:17:17.116271 16829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:17:17.116449 16829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:17:17.122985 16829 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42685
I20260924 04:17:17.123257 16896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42685 every 8 connection(s)
I20260924 04:17:17.124084 16829 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:17:17.128136 16897 sys_catalog.cc:263] Verifying existing consensus state
I20260924 04:17:17.128322 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 16829
I20260924 04:17:17.128697 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37857
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:42685
--webserver_port=44351
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:17:17.129772 16897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac: Bootstrap starting.
I20260924 04:17:17.152199 16897 log.cc:824] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac: Log is configured to *not* fsync() on all Append() calls
I20260924 04:17:17.166289 16897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:17:17.166719 16897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac: Bootstrap complete.
I20260924 04:17:17.172065 16897 raft_consensus.cc:348] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a03a0021d8a04239b1acb304a66797ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42685 } }
I20260924 04:17:17.172564 16897 raft_consensus.cc:729] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a03a0021d8a04239b1acb304a66797ac, State: Initialized, Role: FOLLOWER
I20260924 04:17:17.173132 16897 consensus_queue.cc:260] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a03a0021d8a04239b1acb304a66797ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42685 } }
I20260924 04:17:17.173275 16897 raft_consensus.cc:388] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:17:17.173341 16897 raft_consensus.cc:482] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:17:17.173434 16897 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac [term 2 FOLLOWER]: Advancing to term 3
I20260924 04:17:17.175928 16897 raft_consensus.cc:504] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a03a0021d8a04239b1acb304a66797ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42685 } }
I20260924 04:17:17.176311 16897 leader_election.cc:302] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a03a0021d8a04239b1acb304a66797ac; no voters:
I20260924 04:17:17.176659 16897 leader_election.cc:288] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 04:17:17.176776 16901 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac [term 3 FOLLOWER]: Leader election won for term 3
I20260924 04:17:17.177016 16901 raft_consensus.cc:686] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac [term 3 LEADER]: Becoming Leader. State: Replica: a03a0021d8a04239b1acb304a66797ac, State: Running, Role: LEADER
I20260924 04:17:17.177335 16901 consensus_queue.cc:237] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a03a0021d8a04239b1acb304a66797ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42685 } }
I20260924 04:17:17.177937 16897 sys_catalog.cc:564] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:17:17.178702 16903 sys_catalog.cc:455] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader a03a0021d8a04239b1acb304a66797ac. Latest consensus state: current_term: 3 leader_uuid: "a03a0021d8a04239b1acb304a66797ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a03a0021d8a04239b1acb304a66797ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42685 } } }
I20260924 04:17:17.178738 16902 sys_catalog.cc:455] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "a03a0021d8a04239b1acb304a66797ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a03a0021d8a04239b1acb304a66797ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42685 } } }
I20260924 04:17:17.178834 16902 sys_catalog.cc:458] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac [sys.catalog]: This master's current role is: LEADER
I20260924 04:17:17.178834 16903 sys_catalog.cc:458] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac [sys.catalog]: This master's current role is: LEADER
I20260924 04:17:17.179255 16910 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:17:17.184307 16910 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:17:17.184952 16910 catalog_manager.cc:1259] Loaded cluster ID: 7e4a25d231f7467a98226bbbb4df0995
I20260924 04:17:17.185009 16910 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:17:17.186913 16910 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:17:17.187601 16910 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac: Loaded TSK: 0
I20260924 04:17:17.187673 16910 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a03a0021d8a04239b1acb304a66797ac: Loaded TSK: 1
F20260924 04:17:17.188262 16910 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fea320949ad google::LogMessage::Fail() at ??:0
@ 0x7fea3209788a google::LogMessage::SendToLog() at ??:0
@ 0x7fea320944de google::LogMessage::Flush() at ??:0
@ 0x7fea32095a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fea34d5818d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fea34d50f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fea3c9fd7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fea3c9d799c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fea3c9d8178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7fea3ca19df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fea3ca3882b std::function<>::operator()() at ??:0
@ 0x7fea3c9d7b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fea3c9d8f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fea3c9d5579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fea3ca19640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fea342e63d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fea342e6c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fea342e83f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fea342d7512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fea3591b6db start_thread at ??:0
@ 0x7fea30b9261f clone at ??:0
W20260924 04:17:17.287652 16899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:17:17.287901 16899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:17:17.287948 16899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:17:17.292085 16899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:17:17.292141 16899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:17:17.292205 16899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:17:17.297564 16899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37857
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=44351
--tserver_master_addrs=127.0.98.62:42685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:17:17.298614 16899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:17:17.299649 16899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:17:17.309233 16944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:17:17.309250 16947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:17:17.309257 16945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:17:17.309629 16899 server_base.cc:1034] running on GCE node
I20260924 04:17:17.310072 16899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:17:17.310837 16899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:17:17.312026 16899 hybrid_clock.cc:648] HybridClock initialized: now 1790223437311994 us; error 41 us; skew 500 ppm
Sep 24 04:17:17 dist-test-slave-p401 krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:17:17.315160 16899 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:17:17.316208 16899 webserver.cc:460] Webserver started at http://127.0.98.1:44351/ using document root <none> and password file <none>
I20260924 04:17:17.316907 16899 fs_manager.cc:359] Metadata directory not provided
I20260924 04:17:17.316999 16899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:17:17.321841 16899 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.005s
I20260924 04:17:17.324311 16954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:17.325683 16899 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:17:17.325801 16899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-0/wal
uuid: "7a632792b0b5464bb290c8420560401d"
format_stamp: "Formatted at 2026-09-24 04:16:35 on dist-test-slave-p401"
server_key: "e6425b62388f47ff1a883d17ffe9191d"
server_key_iv: "dc187a9f40f73ced6e2a5aed30e12197"
server_key_version: "encryptionkey@0"
I20260924 04:17:17.326471 16899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:17:17.366758 16899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:17:17.370519 16899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:17:17.370711 16899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:17:17.371419 16899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:17:17.372495 16961 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 04:17:17.377940 16899 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 04:17:17.378021 16899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260924 04:17:17.378091 16899 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 04:17:17.379918 16899 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 04:17:17.379971 16899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260924 04:17:17.380322 16961 tablet_bootstrap.cc:492] T 115649d93b6c45f4a3db817b32b4ca62 P 7a632792b0b5464bb290c8420560401d: Bootstrap starting.
I20260924 04:17:17.389768 16899 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37857
I20260924 04:17:17.389801 17068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37857 every 8 connection(s)
I20260924 04:17:17.390779 16899 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:17:17.396248 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 16899
I20260924 04:17:17.396903 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:44431
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:42685
--webserver_port=45289
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:17:17.402863 16961 log.cc:824] T 115649d93b6c45f4a3db817b32b4ca62 P 7a632792b0b5464bb290c8420560401d: Log is configured to *not* fsync() on all Append() calls
I20260924 04:17:17.410193 16961 tablet_bootstrap.cc:492] T 115649d93b6c45f4a3db817b32b4ca62 P 7a632792b0b5464bb290c8420560401d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:17:17.410583 16961 tablet_bootstrap.cc:492] T 115649d93b6c45f4a3db817b32b4ca62 P 7a632792b0b5464bb290c8420560401d: Bootstrap complete.
I20260924 04:17:17.411558 16961 ts_tablet_manager.cc:1397] T 115649d93b6c45f4a3db817b32b4ca62 P 7a632792b0b5464bb290c8420560401d: Time spent bootstrapping tablet: real 0.032s user 0.022s sys 0.008s
I20260924 04:17:17.414003 16961 raft_consensus.cc:348] T 115649d93b6c45f4a3db817b32b4ca62 P 7a632792b0b5464bb290c8420560401d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adfc52788f744448b30368cba1c1cd5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44431 } } peers { permanent_uuid: "d8b0a26a9a4141fb95452f16af8283fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } } peers { permanent_uuid: "7a632792b0b5464bb290c8420560401d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37857 } }
I20260924 04:17:17.414458 16961 raft_consensus.cc:729] T 115649d93b6c45f4a3db817b32b4ca62 P 7a632792b0b5464bb290c8420560401d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a632792b0b5464bb290c8420560401d, State: Initialized, Role: FOLLOWER
I20260924 04:17:17.414943 16961 consensus_queue.cc:260] T 115649d93b6c45f4a3db817b32b4ca62 P 7a632792b0b5464bb290c8420560401d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adfc52788f744448b30368cba1c1cd5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44431 } } peers { permanent_uuid: "d8b0a26a9a4141fb95452f16af8283fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } } peers { permanent_uuid: "7a632792b0b5464bb290c8420560401d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37857 } }
I20260924 04:17:17.415912 16961 ts_tablet_manager.cc:1428] T 115649d93b6c45f4a3db817b32b4ca62 P 7a632792b0b5464bb290c8420560401d: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
W20260924 04:17:17.422915 17071 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:17:17.392808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.1:38709)
0924 04:17:17.394020 (+ 1212us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:17.394085 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 04:17:17.395172 (+ 1087us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:17:17.421944 (+ 26772us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42685: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":133,"threads_started":1}
W20260924 04:17:17.423460 17069 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42685 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:42685: Client connection negotiation failed: client connection to 127.0.98.62:42685: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 04:17:17.424175 17071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:17.423900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.1:59719)
0924 04:17:17.424031 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:17.424109 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 04:17:17.424830 17071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:17.424630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.1:51915)
0924 04:17:17.424726 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:17.424776 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 04:17:17.425031 17069 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 04:17:17.513166 17072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:17:17.513433 17072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:17:17.513477 17072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:17:17.517453 17072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:17:17.517514 17072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:17:17.517573 17072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:17:17.522775 17072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:44431
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=45289
--tserver_master_addrs=127.0.98.62:42685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:17:17.523851 17072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:17:17.524853 17072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:17:17.534327 17079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:17:17.534382 17072 server_base.cc:1034] running on GCE node
W20260924 04:17:17.534328 17082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:17:17.534327 17080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:17:17.534898 17072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:17:17.535573 17072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:17:17.536731 17072 hybrid_clock.cc:648] HybridClock initialized: now 1790223437536711 us; error 32 us; skew 500 ppm
Sep 24 04:17:17 dist-test-slave-p401 krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:17:17.539353 17072 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:17:17.540201 17072 webserver.cc:460] Webserver started at http://127.0.98.2:45289/ using document root <none> and password file <none>
I20260924 04:17:17.540710 17072 fs_manager.cc:359] Metadata directory not provided
I20260924 04:17:17.540766 17072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:17:17.544663 17072 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:17:17.546794 17089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:17.548007 17072 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:17:17.548100 17072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-1/wal
uuid: "adfc52788f744448b30368cba1c1cd5e"
format_stamp: "Formatted at 2026-09-24 04:16:35 on dist-test-slave-p401"
server_key: "7c2671fb1614616197695f60d2825b1e"
server_key_iv: "9ee2272ba04d60a708429de355b63914"
server_key_version: "encryptionkey@0"
I20260924 04:17:17.548626 17072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:17:17.564127 17072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:17:17.566799 17072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:17:17.566944 17072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:17:17.567487 17072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:17:17.568460 17096 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 04:17:17.572309 17072 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 04:17:17.572358 17072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260924 04:17:17.572403 17072 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 04:17:17.573926 17072 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 04:17:17.573966 17072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260924 04:17:17.574219 17096 tablet_bootstrap.cc:492] T 115649d93b6c45f4a3db817b32b4ca62 P adfc52788f744448b30368cba1c1cd5e: Bootstrap starting.
I20260924 04:17:17.583055 17072 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44431
I20260924 04:17:17.583141 17203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44431 every 8 connection(s)
I20260924 04:17:17.583977 17072 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:17:17.584200 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 17072
I20260924 04:17:17.584661 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37753
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:42685
--webserver_port=40435
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:17:17.587993 17207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:17.586426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.2:40547)
0924 04:17:17.586927 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:17.587048 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":130,"threads_started":1}
W20260924 04:17:17.588490 17204 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42685 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:42685: Client connection negotiation failed: client connection to 127.0.98.62:42685: connect: Connection refused (error 111)
I20260924 04:17:17.589252 17207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:17.589000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.2:45267)
0924 04:17:17.589133 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:17.589188 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 04:17:17.589890 17207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:17.589672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.2:48939)
0924 04:17:17.589778 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:17.589827 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 04:17:17.590080 17204 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 04:17:17.592000 17096 log.cc:824] T 115649d93b6c45f4a3db817b32b4ca62 P adfc52788f744448b30368cba1c1cd5e: Log is configured to *not* fsync() on all Append() calls
I20260924 04:17:17.598548 17096 tablet_bootstrap.cc:492] T 115649d93b6c45f4a3db817b32b4ca62 P adfc52788f744448b30368cba1c1cd5e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:17:17.598925 17096 tablet_bootstrap.cc:492] T 115649d93b6c45f4a3db817b32b4ca62 P adfc52788f744448b30368cba1c1cd5e: Bootstrap complete.
I20260924 04:17:17.599879 17096 ts_tablet_manager.cc:1397] T 115649d93b6c45f4a3db817b32b4ca62 P adfc52788f744448b30368cba1c1cd5e: Time spent bootstrapping tablet: real 0.026s user 0.010s sys 0.014s
I20260924 04:17:17.602535 17096 raft_consensus.cc:348] T 115649d93b6c45f4a3db817b32b4ca62 P adfc52788f744448b30368cba1c1cd5e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adfc52788f744448b30368cba1c1cd5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44431 } } peers { permanent_uuid: "d8b0a26a9a4141fb95452f16af8283fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } } peers { permanent_uuid: "7a632792b0b5464bb290c8420560401d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37857 } }
I20260924 04:17:17.602946 17096 raft_consensus.cc:729] T 115649d93b6c45f4a3db817b32b4ca62 P adfc52788f744448b30368cba1c1cd5e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: adfc52788f744448b30368cba1c1cd5e, State: Initialized, Role: FOLLOWER
I20260924 04:17:17.603408 17096 consensus_queue.cc:260] T 115649d93b6c45f4a3db817b32b4ca62 P adfc52788f744448b30368cba1c1cd5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adfc52788f744448b30368cba1c1cd5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44431 } } peers { permanent_uuid: "d8b0a26a9a4141fb95452f16af8283fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } } peers { permanent_uuid: "7a632792b0b5464bb290c8420560401d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37857 } }
I20260924 04:17:17.604388 17096 ts_tablet_manager.cc:1428] T 115649d93b6c45f4a3db817b32b4ca62 P adfc52788f744448b30368cba1c1cd5e: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260924 04:17:17.604653 17207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:17.604365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.2:59465)
0924 04:17:17.604498 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:17.604564 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 04:17:17.605290 17207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:17.605079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.2:50853)
0924 04:17:17.605181 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:17.605231 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 04:17:17.692428 17206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:17:17.692689 17206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:17:17.692731 17206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:17:17.696743 17206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:17:17.696797 17206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:17:17.696862 17206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:17:17.702045 17206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37753
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40435
--tserver_master_addrs=127.0.98.62:42685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:17:17.703027 17206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:17:17.704015 17206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:17:17.713364 17215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:17:17.713375 17214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:17:17.713445 17206 server_base.cc:1034] running on GCE node
W20260924 04:17:17.713392 17217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:17:17.713866 17206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:17:17.714483 17206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:17:17.715632 17206 hybrid_clock.cc:648] HybridClock initialized: now 1790223437715603 us; error 41 us; skew 500 ppm
Sep 24 04:17:17 dist-test-slave-p401 krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:17:17.718161 17206 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:17:17.718930 17206 webserver.cc:460] Webserver started at http://127.0.98.3:40435/ using document root <none> and password file <none>
I20260924 04:17:17.719463 17206 fs_manager.cc:359] Metadata directory not provided
I20260924 04:17:17.719523 17206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:17:17.723321 17206 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:17:17.725193 17224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:17.726326 17206 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:17:17.726421 17206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-2/wal
uuid: "d8b0a26a9a4141fb95452f16af8283fc"
format_stamp: "Formatted at 2026-09-24 04:16:35 on dist-test-slave-p401"
server_key: "f7ae8f9b974f75cd6d051bef2100e1d5"
server_key_iv: "a515fd50bab75fbdfdd9f322ea1a8b83"
server_key_version: "encryptionkey@0"
I20260924 04:17:17.726931 17206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:17:17.749441 17206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:17:17.752538 17206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:17:17.752703 17206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:17:17.753264 17206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:17:17.754066 17206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:17:17.754112 17206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:17.754153 17206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:17:17.754180 17206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:17.762470 17206 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37753
I20260924 04:17:17.762511 17337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37753 every 8 connection(s)
I20260924 04:17:17.763466 17206 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:17:17.766970 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:17.765444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.3:57157)
0924 04:17:17.766146 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:17.766261 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":111,"threads_started":1}
W20260924 04:17:17.767400 17338 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42685 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:42685: Client connection negotiation failed: client connection to 127.0.98.62:42685: connect: Connection refused (error 111)
I20260924 04:17:17.768000 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:17.767774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.3:34131)
0924 04:17:17.767897 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:17.767945 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 04:17:17.768553 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:17.768378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.3:41805)
0924 04:17:17.768463 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:17.768503 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 04:17:17.768697 17338 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 04:17:17.771440 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 17206
I20260924 04:17:18.425993 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:18.425591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.1:46549)
0924 04:17:18.425832 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:18.425885 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":69,"threads_started":1}
I20260924 04:17:18.606459 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:18.605979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.2:48653)
0924 04:17:18.606275 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:18.606339 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":82,"threads_started":1}
I20260924 04:17:18.630487 17344 raft_consensus.cc:482] T 115649d93b6c45f4a3db817b32b4ca62 P 7a632792b0b5464bb290c8420560401d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:17:18.630682 17344 raft_consensus.cc:504] T 115649d93b6c45f4a3db817b32b4ca62 P 7a632792b0b5464bb290c8420560401d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adfc52788f744448b30368cba1c1cd5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44431 } } peers { permanent_uuid: "d8b0a26a9a4141fb95452f16af8283fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } } peers { permanent_uuid: "7a632792b0b5464bb290c8420560401d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37857 } }
I20260924 04:17:18.631914 17344 leader_election.cc:288] T 115649d93b6c45f4a3db817b32b4ca62 P 7a632792b0b5464bb290c8420560401d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers adfc52788f744448b30368cba1c1cd5e (127.0.98.2:44431), d8b0a26a9a4141fb95452f16af8283fc (127.0.98.3:37753)
Sep 24 04:17:18 dist-test-slave-p401 krb5kdc[15751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 24 04:17:18 dist-test-slave-p401 krb5kdc[15751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260924 04:17:18.644897 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:18.632287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57687 (local address 127.0.98.3:37753)
0924 04:17:18.632580 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:18.632612 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:18.632713 (+ 101us) server_negotiation.cc:407] Connection header received
0924 04:17:18.633447 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:18.633453 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:18.633759 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:18.634003 (+ 244us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:17:18.635083 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:18.636063 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:18.637264 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:18.637719 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:18.637859 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:18.640942 (+ 3083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:17:18.640947 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:17:18.640959 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:17:18.640992 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:17:18.642517 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:17:18.643040 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:17:18.643045 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:17:18.643049 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:17:18.643155 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:17:18.643496 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:17:18.643499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:17:18.643501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:17:18.643896 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:17:18.644010 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:18.644501 (+ 491us) server_negotiation.cc:299] Negotiation successful
0924 04:17:18.644746 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":118,"threads_started":1}
I20260924 04:17:18.644886 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:18.632103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37753 (local address 127.0.98.1:57687)
0924 04:17:18.632588 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:18.632606 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 04:17:18.632739 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:17:18.633970 (+ 1231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:17:18.633973 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:17:18.634463 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:17:18.634931 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:18.634944 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:18.636195 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:18.636198 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:18.637158 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:18.637167 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:18.637829 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:18.637833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:18.638026 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:18.638713 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:17:18.638726 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:17:18.640754 (+ 2028us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:17:18.642634 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:17:18.642636 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:17:18.642638 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:17:18.642933 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:17:18.643261 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:17:18.643264 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:17:18.643266 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:17:18.643408 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:17:18.644005 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:17:18.644011 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:17:18.644219 (+ 208us) client_negotiation.cc:769] Sending connection context
0924 04:17:18.644379 (+ 160us) client_negotiation.cc:240] Negotiation successful
0924 04:17:18.644694 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22,"mutex_wait_us":139,"spinlock_wait_cycles":59264}
I20260924 04:17:18.645507 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:18.632103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44431 (local address 127.0.98.1:35719)
0924 04:17:18.632900 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:18.632911 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:17:18.633026 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:17:18.633933 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:17:18.633939 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:17:18.634463 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:17:18.634931 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:18.634944 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:18.636187 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:18.636190 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:18.637160 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:18.637169 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:18.637829 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:18.637832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:18.638030 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:18.638682 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:17:18.638703 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:17:18.640432 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:17:18.642583 (+ 2151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:17:18.642589 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:17:18.642603 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:17:18.642934 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:17:18.643274 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:17:18.643277 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:17:18.643279 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:17:18.643419 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:17:18.645054 (+ 1635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:17:18.645057 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:17:18.645140 (+ 83us) client_negotiation.cc:769] Sending connection context
0924 04:17:18.645228 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 04:17:18.645353 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"mutex_wait_us":629,"spinlock_wait_cycles":7936,"thread_start_us":61,"threads_started":1}
I20260924 04:17:18.645881 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:18.632325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35719 (local address 127.0.98.2:44431)
0924 04:17:18.632826 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:18.632839 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:18.633045 (+ 206us) server_negotiation.cc:407] Connection header received
0924 04:17:18.633267 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:18.633272 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:18.633594 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:18.633862 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:17:18.635094 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:18.636055 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:18.637273 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:18.637719 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:18.637859 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:18.640651 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:17:18.640658 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:17:18.640677 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:17:18.640711 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:17:18.642468 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:17:18.643042 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:17:18.643046 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:17:18.643051 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:17:18.643165 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:17:18.643526 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:17:18.643529 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:17:18.643531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:17:18.643895 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:17:18.644008 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:18.645377 (+ 1369us) server_negotiation.cc:299] Negotiation successful
0924 04:17:18.645765 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":87,"threads_started":1}
I20260924 04:17:18.646849 17158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "115649d93b6c45f4a3db817b32b4ca62" candidate_uuid: "7a632792b0b5464bb290c8420560401d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "adfc52788f744448b30368cba1c1cd5e" is_pre_election: true
I20260924 04:17:18.647181 17158 raft_consensus.cc:2454] T 115649d93b6c45f4a3db817b32b4ca62 P adfc52788f744448b30368cba1c1cd5e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a632792b0b5464bb290c8420560401d in term 1.
W20260924 04:17:18.647532 16956 leader_election.cc:341] T 115649d93b6c45f4a3db817b32b4ca62 P 7a632792b0b5464bb290c8420560401d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d8b0a26a9a4141fb95452f16af8283fc (127.0.98.3:37753): Not found: Tablet not found: 115649d93b6c45f4a3db817b32b4ca62
I20260924 04:17:18.647931 16958 leader_election.cc:302] T 115649d93b6c45f4a3db817b32b4ca62 P 7a632792b0b5464bb290c8420560401d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a632792b0b5464bb290c8420560401d, adfc52788f744448b30368cba1c1cd5e; no voters: d8b0a26a9a4141fb95452f16af8283fc
I20260924 04:17:18.648176 17344 raft_consensus.cc:2781] T 115649d93b6c45f4a3db817b32b4ca62 P 7a632792b0b5464bb290c8420560401d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 04:17:18.648267 17344 raft_consensus.cc:482] T 115649d93b6c45f4a3db817b32b4ca62 P 7a632792b0b5464bb290c8420560401d [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:17:18.648311 17344 raft_consensus.cc:3037] T 115649d93b6c45f4a3db817b32b4ca62 P 7a632792b0b5464bb290c8420560401d [term 1 FOLLOWER]: Advancing to term 2
I20260924 04:17:18.650424 17344 raft_consensus.cc:504] T 115649d93b6c45f4a3db817b32b4ca62 P 7a632792b0b5464bb290c8420560401d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adfc52788f744448b30368cba1c1cd5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44431 } } peers { permanent_uuid: "d8b0a26a9a4141fb95452f16af8283fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } } peers { permanent_uuid: "7a632792b0b5464bb290c8420560401d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37857 } }
I20260924 04:17:18.650743 17344 leader_election.cc:288] T 115649d93b6c45f4a3db817b32b4ca62 P 7a632792b0b5464bb290c8420560401d [CANDIDATE]: Term 2 election: Requested vote from peers adfc52788f744448b30368cba1c1cd5e (127.0.98.2:44431), d8b0a26a9a4141fb95452f16af8283fc (127.0.98.3:37753)
I20260924 04:17:18.651089 17158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "115649d93b6c45f4a3db817b32b4ca62" candidate_uuid: "7a632792b0b5464bb290c8420560401d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "adfc52788f744448b30368cba1c1cd5e"
I20260924 04:17:18.651252 17158 raft_consensus.cc:3037] T 115649d93b6c45f4a3db817b32b4ca62 P adfc52788f744448b30368cba1c1cd5e [term 1 FOLLOWER]: Advancing to term 2
W20260924 04:17:18.651404 16956 leader_election.cc:341] T 115649d93b6c45f4a3db817b32b4ca62 P 7a632792b0b5464bb290c8420560401d [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer d8b0a26a9a4141fb95452f16af8283fc (127.0.98.3:37753): Not found: Tablet not found: 115649d93b6c45f4a3db817b32b4ca62
I20260924 04:17:18.653311 17158 raft_consensus.cc:2454] T 115649d93b6c45f4a3db817b32b4ca62 P adfc52788f744448b30368cba1c1cd5e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a632792b0b5464bb290c8420560401d in term 2.
I20260924 04:17:18.653553 16958 leader_election.cc:302] T 115649d93b6c45f4a3db817b32b4ca62 P 7a632792b0b5464bb290c8420560401d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a632792b0b5464bb290c8420560401d, adfc52788f744448b30368cba1c1cd5e; no voters: d8b0a26a9a4141fb95452f16af8283fc
I20260924 04:17:18.653703 17344 raft_consensus.cc:2781] T 115649d93b6c45f4a3db817b32b4ca62 P 7a632792b0b5464bb290c8420560401d [term 2 FOLLOWER]: Leader election won for term 2
I20260924 04:17:18.653856 17344 raft_consensus.cc:686] T 115649d93b6c45f4a3db817b32b4ca62 P 7a632792b0b5464bb290c8420560401d [term 2 LEADER]: Becoming Leader. State: Replica: 7a632792b0b5464bb290c8420560401d, State: Running, Role: LEADER
I20260924 04:17:18.654141 17344 consensus_queue.cc:237] T 115649d93b6c45f4a3db817b32b4ca62 P 7a632792b0b5464bb290c8420560401d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adfc52788f744448b30368cba1c1cd5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44431 } } peers { permanent_uuid: "d8b0a26a9a4141fb95452f16af8283fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } } peers { permanent_uuid: "7a632792b0b5464bb290c8420560401d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37857 } }
I20260924 04:17:18.654481 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:18.654279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.1:58449)
0924 04:17:18.654377 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:18.654424 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 04:17:18.769598 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:18.769183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.3:49257)
0924 04:17:18.769442 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:18.769496 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":79,"threads_started":1}
I20260924 04:17:19.039368 17158 raft_consensus.cc:1261] T 115649d93b6c45f4a3db817b32b4ca62 P adfc52788f744448b30368cba1c1cd5e [term 2 FOLLOWER]: Refusing update from remote peer 7a632792b0b5464bb290c8420560401d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260924 04:17:19.040136 17344 consensus_queue.cc:1035] T 115649d93b6c45f4a3db817b32b4ca62 P 7a632792b0b5464bb290c8420560401d [LEADER]: Connected to new peer: Peer: permanent_uuid: "adfc52788f744448b30368cba1c1cd5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 04:17:19.042038 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:19.041769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.2:44541)
0924 04:17:19.041885 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:19.041944 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 04:17:19.044281 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:19.044015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.1:54415)
0924 04:17:19.044169 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:19.044216 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260924 04:17:19.044492 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:19.044267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.2:37873)
0924 04:17:19.044377 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:19.044427 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 04:17:19.044631 16956 consensus_peers.cc:487] T 115649d93b6c45f4a3db817b32b4ca62 P 7a632792b0b5464bb290c8420560401d -> Peer d8b0a26a9a4141fb95452f16af8283fc (127.0.98.3:37753): Couldn't send request to peer d8b0a26a9a4141fb95452f16af8283fc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 115649d93b6c45f4a3db817b32b4ca62. This is attempt 1: this message will repeat every 5th retry.
I20260924 04:17:19.561728 17357 ts_tablet_manager.cc:927] T 115649d93b6c45f4a3db817b32b4ca62 P d8b0a26a9a4141fb95452f16af8283fc: Initiating tablet copy from peer 7a632792b0b5464bb290c8420560401d (127.0.98.1:37857)
I20260924 04:17:19.562282 17357 tablet_copy_client.cc:316] T 115649d93b6c45f4a3db817b32b4ca62 P d8b0a26a9a4141fb95452f16af8283fc: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.1:37857
Sep 24 04:17:19 dist-test-slave-p401 krb5kdc[15751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:17:19.572084 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:19.562758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37857 (local address 127.0.98.3:51461)
0924 04:17:19.562991 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:19.563012 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 04:17:19.563151 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:17:19.563642 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:17:19.563647 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:17:19.564038 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:17:19.564152 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:19.564158 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:19.565109 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:19.565112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:19.565777 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:19.565786 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:19.566353 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:19.566356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:19.566462 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:19.567109 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:17:19.567122 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:17:19.568509 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:17:19.570406 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:17:19.570413 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:17:19.570418 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:17:19.570636 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:17:19.570908 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:17:19.570912 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:17:19.570914 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:17:19.570985 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:17:19.571464 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:17:19.571471 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:17:19.571590 (+ 119us) client_negotiation.cc:769] Sending connection context
0924 04:17:19.571727 (+ 137us) client_negotiation.cc:240] Negotiation successful
0924 04:17:19.571896 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":97,"threads_started":1}
I20260924 04:17:19.572119 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:19.562912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51461 (local address 127.0.98.1:37857)
0924 04:17:19.563201 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:19.563208 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:19.563257 (+ 49us) server_negotiation.cc:407] Connection header received
0924 04:17:19.563304 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:19.563308 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:19.563485 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:19.563609 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:17:19.564269 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:19.565003 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:19.565885 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:19.566245 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:19.566336 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:19.568637 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:17:19.568645 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:17:19.568658 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:17:19.568685 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:17:19.570305 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:17:19.570743 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:17:19.570748 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:17:19.570750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:17:19.570827 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:17:19.571063 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:17:19.571066 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:17:19.571068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:17:19.571362 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:17:19.571484 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:19.571836 (+ 352us) server_negotiation.cc:299] Negotiation successful
0924 04:17:19.571981 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":76,"threads_started":1}
I20260924 04:17:19.573192 17043 tablet_copy_service.cc:140] P 7a632792b0b5464bb290c8420560401d: Received BeginTabletCopySession request for tablet 115649d93b6c45f4a3db817b32b4ca62 from peer d8b0a26a9a4141fb95452f16af8283fc ({username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51461)
I20260924 04:17:19.573341 17043 tablet_copy_service.cc:161] P 7a632792b0b5464bb290c8420560401d: Beginning new tablet copy session on tablet 115649d93b6c45f4a3db817b32b4ca62 from peer d8b0a26a9a4141fb95452f16af8283fc at {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51461: session id = d8b0a26a9a4141fb95452f16af8283fc-115649d93b6c45f4a3db817b32b4ca62
I20260924 04:17:19.574470 17043 tablet_copy_source_session.cc:205] T 115649d93b6c45f4a3db817b32b4ca62 P 7a632792b0b5464bb290c8420560401d: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 04:17:19.578270 17357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 115649d93b6c45f4a3db817b32b4ca62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:17:19.584877 17357 tablet_copy_client.cc:791] T 115649d93b6c45f4a3db817b32b4ca62 P d8b0a26a9a4141fb95452f16af8283fc: tablet copy: Starting download of 0 data blocks...
I20260924 04:17:19.585122 17357 tablet_copy_client.cc:655] T 115649d93b6c45f4a3db817b32b4ca62 P d8b0a26a9a4141fb95452f16af8283fc: tablet copy: Starting download of 1 WAL segments...
I20260924 04:17:19.586884 17357 tablet_copy_client.cc:531] T 115649d93b6c45f4a3db817b32b4ca62 P d8b0a26a9a4141fb95452f16af8283fc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 04:17:19.588691 17357 tablet_bootstrap.cc:492] T 115649d93b6c45f4a3db817b32b4ca62 P d8b0a26a9a4141fb95452f16af8283fc: Bootstrap starting.
I20260924 04:17:19.593976 17357 log.cc:824] T 115649d93b6c45f4a3db817b32b4ca62 P d8b0a26a9a4141fb95452f16af8283fc: Log is configured to *not* fsync() on all Append() calls
I20260924 04:17:19.600324 17357 tablet_bootstrap.cc:492] T 115649d93b6c45f4a3db817b32b4ca62 P d8b0a26a9a4141fb95452f16af8283fc: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:17:19.600629 17357 tablet_bootstrap.cc:492] T 115649d93b6c45f4a3db817b32b4ca62 P d8b0a26a9a4141fb95452f16af8283fc: Bootstrap complete.
I20260924 04:17:19.600934 17357 ts_tablet_manager.cc:1397] T 115649d93b6c45f4a3db817b32b4ca62 P d8b0a26a9a4141fb95452f16af8283fc: Time spent bootstrapping tablet: real 0.012s user 0.012s sys 0.000s
I20260924 04:17:19.603221 17357 raft_consensus.cc:348] T 115649d93b6c45f4a3db817b32b4ca62 P d8b0a26a9a4141fb95452f16af8283fc [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adfc52788f744448b30368cba1c1cd5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44431 } } peers { permanent_uuid: "d8b0a26a9a4141fb95452f16af8283fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } } peers { permanent_uuid: "7a632792b0b5464bb290c8420560401d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37857 } }
I20260924 04:17:19.603508 17357 raft_consensus.cc:729] T 115649d93b6c45f4a3db817b32b4ca62 P d8b0a26a9a4141fb95452f16af8283fc [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8b0a26a9a4141fb95452f16af8283fc, State: Initialized, Role: FOLLOWER
I20260924 04:17:19.604027 17357 consensus_queue.cc:260] T 115649d93b6c45f4a3db817b32b4ca62 P d8b0a26a9a4141fb95452f16af8283fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adfc52788f744448b30368cba1c1cd5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44431 } } peers { permanent_uuid: "d8b0a26a9a4141fb95452f16af8283fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } } peers { permanent_uuid: "7a632792b0b5464bb290c8420560401d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37857 } }
I20260924 04:17:19.604893 17357 ts_tablet_manager.cc:1428] T 115649d93b6c45f4a3db817b32b4ca62 P d8b0a26a9a4141fb95452f16af8283fc: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260924 04:17:19.605163 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:19.604915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.3:35629)
0924 04:17:19.605030 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:19.605088 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 04:17:19.605562 17043 tablet_copy_service.cc:342] P 7a632792b0b5464bb290c8420560401d: Request end of tablet copy session d8b0a26a9a4141fb95452f16af8283fc-115649d93b6c45f4a3db817b32b4ca62 received from {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51461
I20260924 04:17:19.605666 17043 tablet_copy_service.cc:433] P 7a632792b0b5464bb290c8420560401d: ending tablet copy session d8b0a26a9a4141fb95452f16af8283fc-115649d93b6c45f4a3db817b32b4ca62 on tablet 115649d93b6c45f4a3db817b32b4ca62 with peer d8b0a26a9a4141fb95452f16af8283fc
I20260924 04:17:19.605786 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:19.605574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.3:60685)
0924 04:17:19.605676 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:19.605725 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 04:17:20.045423 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:20.044987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.1:37449)
0924 04:17:20.045265 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:20.045328 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":93,"threads_started":1}
I20260924 04:17:20.045506 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:20.045160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.2:38579)
0924 04:17:20.045366 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:20.045417 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":58,"threads_started":1}
I20260924 04:17:20.064754 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:20.064503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.1:52795)
0924 04:17:20.064621 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:20.064674 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 04:17:20.064898 17292 raft_consensus.cc:1203] T 115649d93b6c45f4a3db817b32b4ca62 P d8b0a26a9a4141fb95452f16af8283fc [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260924 04:17:20.065891 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:20.065666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.3:54763)
0924 04:17:20.065780 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:20.065828 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 04:17:21.046727 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:21.046235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.2:59521)
0924 04:17:21.046536 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:21.046600 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":66,"threads_started":1}
I20260924 04:17:21.065908 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:21.065485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.1:41597)
0924 04:17:21.065752 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:21.065815 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":84,"threads_started":1}
I20260924 04:17:21.066877 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:21.066504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.3:40817)
0924 04:17:21.066739 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:21.066794 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":75,"threads_started":1}
I20260924 04:17:22.047961 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:22.047520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.2:35641)
0924 04:17:22.047794 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:22.047860 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":94,"threads_started":1}
I20260924 04:17:22.066923 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:22.066538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.1:54951)
0924 04:17:22.066782 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:22.066835 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":87,"threads_started":1}
I20260924 04:17:22.067850 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:22.067468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.3:37015)
0924 04:17:22.067724 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:22.067776 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":65,"threads_started":1}
I20260924 04:17:23.049252 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:23.048809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.2:46471)
0924 04:17:23.049095 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:23.049153 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":75,"threads_started":1}
I20260924 04:17:23.068297 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:23.067843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.1:32867)
0924 04:17:23.068115 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:23.068181 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":84,"threads_started":1}
I20260924 04:17:23.069008 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:23.068613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.3:43987)
0924 04:17:23.068867 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:23.068928 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":73,"threads_started":1}
I20260924 04:17:23.068996 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:23.068767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.1:43347)
0924 04:17:23.068884 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:23.068927 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 04:17:24.050467 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:24.050041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.2:52017)
0924 04:17:24.050307 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:24.050368 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":83,"threads_started":1}
I20260924 04:17:24.070156 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:24.069721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.3:35593)
0924 04:17:24.069995 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:24.070060 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":84,"threads_started":1}
I20260924 04:17:24.070199 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:24.069726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.1:58813)
0924 04:17:24.070046 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:24.070098 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":87,"threads_started":1}
I20260924 04:17:25.051854 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:25.051392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.2:59359)
0924 04:17:25.051652 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:25.051732 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":67,"threads_started":1}
I20260924 04:17:25.071348 17379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:25.070859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.1:36739)
0924 04:17:25.071137 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:25.071204 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":72,"threads_started":1}
I20260924 04:17:25.071561 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:25.070883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.3:54639)
0924 04:17:25.071391 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:25.071450 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":59,"threads_started":1}
I20260924 04:17:26.052992 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:26.052613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.2:48395)
0924 04:17:26.052838 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:26.052893 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":70,"threads_started":1}
I20260924 04:17:26.072592 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:26.072182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.1:48305)
0924 04:17:26.072426 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:26.072494 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":78,"threads_started":1}
I20260924 04:17:26.072896 17383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:26.072298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.3:56761)
0924 04:17:26.072552 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:26.072598 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":55,"threads_started":1}
I20260924 04:17:27.054112 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:27.053694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.2:40965)
0924 04:17:27.053951 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:27.054016 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":74,"threads_started":1}
I20260924 04:17:27.073786 17385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:27.073346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.1:38611)
0924 04:17:27.073626 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:27.073691 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":88,"threads_started":1}
I20260924 04:17:27.073951 17386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:27.073589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.3:34361)
0924 04:17:27.073795 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:27.073848 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":55,"threads_started":1}
I20260924 04:17:28.055281 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:28.054876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.2:51675)
0924 04:17:28.055097 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:28.055149 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":69,"threads_started":1}
I20260924 04:17:28.074978 17391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:28.074569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.1:39177)
0924 04:17:28.074813 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:28.074876 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":66,"threads_started":1}
I20260924 04:17:28.075039 17392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:28.074690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.3:38309)
0924 04:17:28.074910 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:28.074957 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":73,"threads_started":1}
I20260924 04:17:29.056488 17393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:29.056065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.2:34887)
0924 04:17:29.056311 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:29.056368 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":72,"threads_started":1}
I20260924 04:17:29.076118 17394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:29.075724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.3:53209)
0924 04:17:29.075966 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:29.076025 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":74,"threads_started":1}
I20260924 04:17:29.076165 17395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:29.075724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.1:36849)
0924 04:17:29.076015 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:29.076063 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":77,"threads_started":1}
I20260924 04:17:30.057598 17396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:30.057188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.2:43479)
0924 04:17:30.057426 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:30.057486 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":73,"threads_started":1}
I20260924 04:17:30.077236 17398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:30.076815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.3:54065)
0924 04:17:30.077072 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:30.077127 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":84,"threads_started":1}
I20260924 04:17:30.077275 17397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:30.076821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.1:49939)
0924 04:17:30.077137 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:30.077190 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":88,"threads_started":1}
I20260924 04:17:31.058732 17399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:31.058291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.2:60193)
0924 04:17:31.058581 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:31.058636 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":83,"threads_started":1}
I20260924 04:17:31.078280 17400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:31.077909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.3:49113)
0924 04:17:31.078132 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:31.078185 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":67,"threads_started":1}
I20260924 04:17:31.078394 17401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:31.077997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.1:41955)
0924 04:17:31.078249 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:31.078299 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":100,"threads_started":1}
I20260924 04:17:32.059943 17402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:32.059474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.2:45569)
0924 04:17:32.059770 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:32.059835 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":85,"threads_started":1}
I20260924 04:17:32.079492 17403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:32.079050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.3:40587)
0924 04:17:32.079329 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:32.079395 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":99,"threads_started":1}
I20260924 04:17:32.079532 17404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:32.079050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42685 (local address 127.0.98.1:47737)
0924 04:17:32.079399 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:32.079449 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":101,"threads_started":1}
I20260924 04:17:32.775112 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 16899
I20260924 04:17:32.780088 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 17072
I20260924 04:17:32.785704 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 17206
2026-09-24T04:17:32Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58983 ms)
[----------] 30 tests from SecurityITest (236511 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:17:32 dist-test-slave-p401 krb5kdc[17407](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:17:32 dist-test-slave-p401 krb5kdc[17407](info): set up 2 sockets
Sep 24 04:17:32 dist-test-slave-p401 krb5kdc[17407](info): commencing operation
krb5kdc: starting...
W20260924 04:17:33.738793 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/krb5kdc/test-user.keytab.
Sep 24 04:17:33 dist-test-slave-p401 krb5kdc[17407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:17:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:17:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:17:33.855556 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36411
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36411
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:17:33.963796 17420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:17:33.964157 17420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:17:33.964221 17420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:17:33.968389 17420 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:17:33.968470 17420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:17:33.968497 17420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:17:33.968518 17420 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:17:33.968539 17420 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:17:33.974088 17420 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:17:33.975267 17420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:17:33.976353 17420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:17:33.984889 17420 server_base.cc:1034] running on GCE node
W20260924 04:17:33.984843 17428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:17:33.984843 17426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:17:33.984869 17425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:17:33.985610 17420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:17:33.986689 17420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:17:33.987849 17420 hybrid_clock.cc:648] HybridClock initialized: now 1790223453987832 us; error 32 us; skew 500 ppm
Sep 24 04:17:33 dist-test-slave-p401 krb5kdc[17407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:17:33.990741 17420 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:17:33.991714 17420 webserver.cc:460] Webserver started at http://127.0.98.62:35273/ using document root <none> and password file <none>
I20260924 04:17:33.992265 17420 fs_manager.cc:359] Metadata directory not provided
I20260924 04:17:33.992328 17420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:17:33.992511 17420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:17:33.994611 17420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/master-0/data/instance:
uuid: "2f35f35222b3419eaf87d8ecce217401"
format_stamp: "Formatted at 2026-09-24 04:17:33 on dist-test-slave-p401"
server_key: "dcb2055ce207b590d4c2eb8154eee5ea"
server_key_iv: "d8d66254ec687833b8155d5ca1cd738f"
server_key_version: "encryptionkey@0"
I20260924 04:17:33.995174 17420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/master-0/wal/instance:
uuid: "2f35f35222b3419eaf87d8ecce217401"
format_stamp: "Formatted at 2026-09-24 04:17:33 on dist-test-slave-p401"
server_key: "dcb2055ce207b590d4c2eb8154eee5ea"
server_key_iv: "d8d66254ec687833b8155d5ca1cd738f"
server_key_version: "encryptionkey@0"
I20260924 04:17:33.998946 17420 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:17:34.001542 17435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:34.002792 17420 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 04:17:34.002890 17420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/master-0/wal
uuid: "2f35f35222b3419eaf87d8ecce217401"
format_stamp: "Formatted at 2026-09-24 04:17:33 on dist-test-slave-p401"
server_key: "dcb2055ce207b590d4c2eb8154eee5ea"
server_key_iv: "d8d66254ec687833b8155d5ca1cd738f"
server_key_version: "encryptionkey@0"
I20260924 04:17:34.002977 17420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:17:34.026144 17420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:17:34.029125 17420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:17:34.029279 17420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:17:34.035948 17420 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36411
I20260924 04:17:34.035959 17487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36411 every 8 connection(s)
I20260924 04:17:34.036903 17420 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:17:34.039667 17488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:17:34.041661 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 17420
I20260924 04:17:34.041796 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/master-0/wal/instance
I20260924 04:17:34.042086 392 external_mini_cluster.cc:1428] Setting key f6982f76c82d9fbafee8c1ab7ec4cfc0
I20260924 04:17:34.046262 17488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f35f35222b3419eaf87d8ecce217401: Bootstrap starting.
I20260924 04:17:34.048713 17488 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f35f35222b3419eaf87d8ecce217401: Neither blocks nor log segments found. Creating new log.
I20260924 04:17:34.049536 17488 log.cc:824] T 00000000000000000000000000000000 P 2f35f35222b3419eaf87d8ecce217401: Log is configured to *not* fsync() on all Append() calls
Sep 24 04:17:34 dist-test-slave-p401 krb5kdc[17407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:17:34.051541 17488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f35f35222b3419eaf87d8ecce217401: No bootstrap required, opened a new log
I20260924 04:17:34.055263 17488 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f35f35222b3419eaf87d8ecce217401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f35f35222b3419eaf87d8ecce217401" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36411 } }
I20260924 04:17:34.055488 17488 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f35f35222b3419eaf87d8ecce217401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:17:34.055542 17488 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f35f35222b3419eaf87d8ecce217401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f35f35222b3419eaf87d8ecce217401, State: Initialized, Role: FOLLOWER
I20260924 04:17:34.055681 17491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:34.043465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54216 (local address 127.0.98.62:36411)
0924 04:17:34.043862 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:34.043877 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:34.043912 (+ 35us) server_negotiation.cc:407] Connection header received
0924 04:17:34.044531 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:34.044547 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:34.044882 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:34.045149 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:17:34.045966 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:34.046932 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:34.048077 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:34.048534 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:34.048662 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:34.051312 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:17:34.051317 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:17:34.051330 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:17:34.051361 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:17:34.052875 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:17:34.053373 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:17:34.053378 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:17:34.053383 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:17:34.053499 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:17:34.053766 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:17:34.053768 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:17:34.053769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:17:34.054132 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:17:34.054292 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:34.054535 (+ 243us) server_negotiation.cc:299] Negotiation successful
0924 04:17:34.054806 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":120,"threads_started":1}
I20260924 04:17:34.055975 17488 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f35f35222b3419eaf87d8ecce217401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f35f35222b3419eaf87d8ecce217401" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36411 } }
I20260924 04:17:34.056103 17488 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f35f35222b3419eaf87d8ecce217401 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:17:34.056165 17488 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f35f35222b3419eaf87d8ecce217401 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:17:34.056260 17488 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f35f35222b3419eaf87d8ecce217401 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:17:34.057090 17488 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f35f35222b3419eaf87d8ecce217401 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f35f35222b3419eaf87d8ecce217401" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36411 } }
I20260924 04:17:34.057392 17488 leader_election.cc:302] T 00000000000000000000000000000000 P 2f35f35222b3419eaf87d8ecce217401 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f35f35222b3419eaf87d8ecce217401; no voters:
I20260924 04:17:34.057672 17488 leader_election.cc:288] T 00000000000000000000000000000000 P 2f35f35222b3419eaf87d8ecce217401 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:17:34.057744 17493 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f35f35222b3419eaf87d8ecce217401 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:17:34.057941 17493 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f35f35222b3419eaf87d8ecce217401 [term 1 LEADER]: Becoming Leader. State: Replica: 2f35f35222b3419eaf87d8ecce217401, State: Running, Role: LEADER
I20260924 04:17:34.058254 17493 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f35f35222b3419eaf87d8ecce217401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f35f35222b3419eaf87d8ecce217401" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36411 } }
I20260924 04:17:34.058929 17488 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f35f35222b3419eaf87d8ecce217401 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:17:34.059861 17495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f35f35222b3419eaf87d8ecce217401 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f35f35222b3419eaf87d8ecce217401. Latest consensus state: current_term: 1 leader_uuid: "2f35f35222b3419eaf87d8ecce217401" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f35f35222b3419eaf87d8ecce217401" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36411 } } }
I20260924 04:17:34.059978 17495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f35f35222b3419eaf87d8ecce217401 [sys.catalog]: This master's current role is: LEADER
I20260924 04:17:34.060406 17502 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:17:34.063169 17502 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:17:34.060392 17494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f35f35222b3419eaf87d8ecce217401 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f35f35222b3419eaf87d8ecce217401" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f35f35222b3419eaf87d8ecce217401" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36411 } } }
I20260924 04:17:34.063820 17494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f35f35222b3419eaf87d8ecce217401 [sys.catalog]: This master's current role is: LEADER
I20260924 04:17:34.068329 17502 catalog_manager.cc:1347] Generated new cluster ID: a5b3f5fb9fa245809b77bf1c2af4537b
I20260924 04:17:34.068396 17502 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:17:34.077072 17502 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:17:34.078034 17502 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:17:34.085031 17502 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f35f35222b3419eaf87d8ecce217401: Generated new TSK 0
I20260924 04:17:34.085698 17502 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:17:34.118832 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36411
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:17:34.231442 17514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:17:34.231693 17514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:17:34.231738 17514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:17:34.235823 17514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:17:34.235878 17514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:17:34.235948 17514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:17:34.241142 17514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:17:34.242142 17514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:17:34.243126 17514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:17:34.252542 17520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:17:34.252624 17522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:17:34.252689 17514 server_base.cc:1034] running on GCE node
W20260924 04:17:34.252524 17519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:17:34.253250 17514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:17:34.253839 17514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:17:34.254985 17514 hybrid_clock.cc:648] HybridClock initialized: now 1790223454254967 us; error 36 us; skew 500 ppm
Sep 24 04:17:34 dist-test-slave-p401 krb5kdc[17407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:17:34.257534 17514 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:17:34.258366 17514 webserver.cc:460] Webserver started at http://127.0.98.1:32985/ using document root <none> and password file <none>
I20260924 04:17:34.258890 17514 fs_manager.cc:359] Metadata directory not provided
I20260924 04:17:34.258952 17514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:17:34.259162 17514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:17:34.261297 17514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/ts-0/data/instance:
uuid: "13f78c4a044144c08a5fe644c6b998bb"
format_stamp: "Formatted at 2026-09-24 04:17:34 on dist-test-slave-p401"
server_key: "2b14ded196ab0edb583ae7f7b3455ded"
server_key_iv: "e558e8b01454903dc94884f300b4780b"
server_key_version: "encryptionkey@0"
I20260924 04:17:34.261814 17514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/ts-0/wal/instance:
uuid: "13f78c4a044144c08a5fe644c6b998bb"
format_stamp: "Formatted at 2026-09-24 04:17:34 on dist-test-slave-p401"
server_key: "2b14ded196ab0edb583ae7f7b3455ded"
server_key_iv: "e558e8b01454903dc94884f300b4780b"
server_key_version: "encryptionkey@0"
I20260924 04:17:34.265328 17514 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:17:34.267582 17529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:34.268743 17514 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:17:34.268852 17514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/ts-0/wal
uuid: "13f78c4a044144c08a5fe644c6b998bb"
format_stamp: "Formatted at 2026-09-24 04:17:34 on dist-test-slave-p401"
server_key: "2b14ded196ab0edb583ae7f7b3455ded"
server_key_iv: "e558e8b01454903dc94884f300b4780b"
server_key_version: "encryptionkey@0"
I20260924 04:17:34.268940 17514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:17:34.279891 17514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:17:34.282956 17514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:17:34.283103 17514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:17:34.283667 17514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:17:34.284443 17514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:17:34.284488 17514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:34.284526 17514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:17:34.284557 17514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:34.292368 17514 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43351
I20260924 04:17:34.292415 17642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43351 every 8 connection(s)
I20260924 04:17:34.293229 17514 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:17:34.294948 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 17514
I20260924 04:17:34.295078 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:17:34.295418 392 external_mini_cluster.cc:1428] Setting key 013ef4fbbc8124f17210cddd996f77c7
Sep 24 04:17:34 dist-test-slave-p401 krb5kdc[17407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:17:34.306650 17491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:34.294877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54117 (local address 127.0.98.62:36411)
0924 04:17:34.294999 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:34.295004 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:34.296066 (+ 1062us) server_negotiation.cc:407] Connection header received
0924 04:17:34.297111 (+ 1045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:34.297115 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:34.297166 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:34.297236 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:17:34.298660 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:34.299491 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:34.300623 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:34.300975 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:34.301082 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:34.303516 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:17:34.303519 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:17:34.303521 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:17:34.303542 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:17:34.304836 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:17:34.305361 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:17:34.305364 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:17:34.305365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:17:34.305435 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:17:34.305717 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:17:34.305719 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:17:34.305720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:17:34.305883 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:17:34.305954 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:34.306408 (+ 454us) server_negotiation.cc:299] Negotiation successful
0924 04:17:34.306528 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 04:17:34.307499 17645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:34.295393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36411 (local address 127.0.98.1:54117)
0924 04:17:34.295904 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:34.295958 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 04:17:34.296883 (+ 925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:17:34.297412 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:17:34.297422 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:17:34.297868 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:17:34.298496 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:34.298515 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:34.299603 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:34.299606 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:34.300503 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:34.300511 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:34.301132 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:34.301135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:34.301276 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:34.301824 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:17:34.301840 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:17:34.303357 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:17:34.304940 (+ 1583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:17:34.304945 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:17:34.304956 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:17:34.305212 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:17:34.305509 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:17:34.305511 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:17:34.305512 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:17:34.305638 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:17:34.305959 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:17:34.305963 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:17:34.306206 (+ 243us) client_negotiation.cc:769] Sending connection context
0924 04:17:34.306356 (+ 150us) client_negotiation.cc:240] Negotiation successful
0924 04:17:34.306615 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":148,"threads_started":1}
I20260924 04:17:34.308578 17643 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36411
I20260924 04:17:34.308813 17643 heartbeater.cc:464] Registering TS with master...
I20260924 04:17:34.309260 17643 heartbeater.cc:511] Master 127.0.98.62:36411 requested a full tablet report, sending...
I20260924 04:17:34.310822 17452 ts_manager.cc:194] Registered new tserver with Master: 13f78c4a044144c08a5fe644c6b998bb (127.0.98.1:43351)
I20260924 04:17:34.312459 17452 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54117
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:17:34.319396 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36411
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:17:34.426486 17648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:17:34.426753 17648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:17:34.426795 17648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:17:34.430768 17648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:17:34.430828 17648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:17:34.430886 17648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:17:34.436154 17648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:17:34.437119 17648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:17:34.438079 17648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:17:34.447510 17654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:17:34.447510 17653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:17:34.447580 17648 server_base.cc:1034] running on GCE node
W20260924 04:17:34.447520 17656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:17:34.448179 17648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:17:34.448809 17648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:17:34.449968 17648 hybrid_clock.cc:648] HybridClock initialized: now 1790223454449947 us; error 37 us; skew 500 ppm
Sep 24 04:17:34 dist-test-slave-p401 krb5kdc[17407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:17:34.452569 17648 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:17:34.453383 17648 webserver.cc:460] Webserver started at http://127.0.98.2:45493/ using document root <none> and password file <none>
I20260924 04:17:34.453904 17648 fs_manager.cc:359] Metadata directory not provided
I20260924 04:17:34.453966 17648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:17:34.454164 17648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:17:34.456197 17648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/ts-1/data/instance:
uuid: "f3bae83e22174b55a109bdacb492e434"
format_stamp: "Formatted at 2026-09-24 04:17:34 on dist-test-slave-p401"
server_key: "9507e69d59b14a18fa31977c02b6cabc"
server_key_iv: "a27bec1790e59fcf5006fdd67f76cd98"
server_key_version: "encryptionkey@0"
I20260924 04:17:34.456686 17648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f3bae83e22174b55a109bdacb492e434"
format_stamp: "Formatted at 2026-09-24 04:17:34 on dist-test-slave-p401"
server_key: "9507e69d59b14a18fa31977c02b6cabc"
server_key_iv: "a27bec1790e59fcf5006fdd67f76cd98"
server_key_version: "encryptionkey@0"
I20260924 04:17:34.460274 17648 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:17:34.462544 17663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:34.463701 17648 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:17:34.463796 17648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/ts-1/wal
uuid: "f3bae83e22174b55a109bdacb492e434"
format_stamp: "Formatted at 2026-09-24 04:17:34 on dist-test-slave-p401"
server_key: "9507e69d59b14a18fa31977c02b6cabc"
server_key_iv: "a27bec1790e59fcf5006fdd67f76cd98"
server_key_version: "encryptionkey@0"
I20260924 04:17:34.463887 17648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:17:34.479755 17648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:17:34.482626 17648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:17:34.482786 17648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:17:34.483348 17648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:17:34.484127 17648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:17:34.484176 17648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:34.484215 17648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:17:34.484234 17648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:34.492204 17648 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41835
I20260924 04:17:34.492237 17776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41835 every 8 connection(s)
I20260924 04:17:34.493019 17648 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:17:34.494899 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 17648
I20260924 04:17:34.494998 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:17:34.495302 392 external_mini_cluster.cc:1428] Setting key bf2dccb7739b6032d01bbd56289ce096
Sep 24 04:17:34 dist-test-slave-p401 krb5kdc[17407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 04:17:34.506305 17491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:34.494655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46301 (local address 127.0.98.62:36411)
0924 04:17:34.494811 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:34.494815 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:34.495711 (+ 896us) server_negotiation.cc:407] Connection header received
0924 04:17:34.496743 (+ 1032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:34.496746 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:34.496798 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:34.496866 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:17:34.498041 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:34.498558 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:34.499847 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:34.500185 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:34.500291 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:34.502873 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:17:34.502877 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:17:34.502880 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:17:34.502909 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:17:34.504424 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:17:34.504914 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:17:34.504917 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:17:34.504918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:17:34.504994 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:17:34.505260 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:17:34.505262 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:17:34.505263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:17:34.505420 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:17:34.505486 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:34.506004 (+ 518us) server_negotiation.cc:299] Negotiation successful
0924 04:17:34.506168 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 04:17:34.507124 17779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:34.495064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36411 (local address 127.0.98.2:46301)
0924 04:17:34.495537 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:34.495595 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 04:17:34.496550 (+ 955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:17:34.497022 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:17:34.497029 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:17:34.497360 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:17:34.497886 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:34.497906 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:34.498690 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:34.498693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:34.499729 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:34.499738 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:34.500352 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:34.500356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:34.500509 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:34.501108 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:17:34.501129 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:17:34.502704 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:17:34.504531 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:17:34.504536 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:17:34.504547 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:17:34.504826 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:17:34.505064 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:17:34.505066 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:17:34.505067 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:17:34.505196 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:17:34.505489 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:17:34.505495 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:17:34.505788 (+ 293us) client_negotiation.cc:769] Sending connection context
0924 04:17:34.505943 (+ 155us) client_negotiation.cc:240] Negotiation successful
0924 04:17:34.506251 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":125,"threads_started":1}
I20260924 04:17:34.508673 17777 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36411
I20260924 04:17:34.508894 17777 heartbeater.cc:464] Registering TS with master...
I20260924 04:17:34.509353 17777 heartbeater.cc:511] Master 127.0.98.62:36411 requested a full tablet report, sending...
I20260924 04:17:34.510341 17452 ts_manager.cc:194] Registered new tserver with Master: f3bae83e22174b55a109bdacb492e434 (127.0.98.2:41835)
I20260924 04:17:34.511091 17452 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46301
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:17:34.520378 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36411
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:17:34.628362 17782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:17:34.628624 17782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:17:34.628669 17782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:17:34.632701 17782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:17:34.632767 17782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:17:34.632848 17782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:17:34.638365 17782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:17:34.639457 17782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:17:34.640434 17782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:17:34.649610 17787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:17:34.649621 17790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:17:34.649610 17788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:17:34.649926 17782 server_base.cc:1034] running on GCE node
I20260924 04:17:34.650332 17782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:17:34.650892 17782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:17:34.652077 17782 hybrid_clock.cc:648] HybridClock initialized: now 1790223454652057 us; error 30 us; skew 500 ppm
Sep 24 04:17:34 dist-test-slave-p401 krb5kdc[17407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:17:34.654748 17782 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:17:34.655682 17782 webserver.cc:460] Webserver started at http://127.0.98.3:41175/ using document root <none> and password file <none>
I20260924 04:17:34.656241 17782 fs_manager.cc:359] Metadata directory not provided
I20260924 04:17:34.656306 17782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:17:34.656515 17782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:17:34.658529 17782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/ts-2/data/instance:
uuid: "94591a7836b54cf3b71eb70e48842f9c"
format_stamp: "Formatted at 2026-09-24 04:17:34 on dist-test-slave-p401"
server_key: "daedbe2eafcdeb822c33b3d1865c619b"
server_key_iv: "4a8fa77a7ef062a989b39eb3cc2b2272"
server_key_version: "encryptionkey@0"
I20260924 04:17:34.659116 17782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/ts-2/wal/instance:
uuid: "94591a7836b54cf3b71eb70e48842f9c"
format_stamp: "Formatted at 2026-09-24 04:17:34 on dist-test-slave-p401"
server_key: "daedbe2eafcdeb822c33b3d1865c619b"
server_key_iv: "4a8fa77a7ef062a989b39eb3cc2b2272"
server_key_version: "encryptionkey@0"
I20260924 04:17:34.662854 17782 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:17:34.665294 17797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:34.666534 17782 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:17:34.666640 17782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/ts-2/wal
uuid: "94591a7836b54cf3b71eb70e48842f9c"
format_stamp: "Formatted at 2026-09-24 04:17:34 on dist-test-slave-p401"
server_key: "daedbe2eafcdeb822c33b3d1865c619b"
server_key_iv: "4a8fa77a7ef062a989b39eb3cc2b2272"
server_key_version: "encryptionkey@0"
I20260924 04:17:34.666736 17782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:17:34.687609 17782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:17:34.690312 17782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:17:34.690486 17782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:17:34.691056 17782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:17:34.691905 17782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:17:34.691952 17782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:34.692005 17782 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:17:34.692077 17782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:34.700070 17782 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42569
I20260924 04:17:34.700095 17910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42569 every 8 connection(s)
I20260924 04:17:34.700912 17782 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:17:34.705581 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 17782
I20260924 04:17:34.705682 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:17:34.705917 392 external_mini_cluster.cc:1428] Setting key f0c7940485e7c1a8061999fbac764bb1
Sep 24 04:17:34 dist-test-slave-p401 krb5kdc[17407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:17:34.712667 17491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:34.702540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33039 (local address 127.0.98.62:36411)
0924 04:17:34.702662 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:34.702666 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:34.703435 (+ 769us) server_negotiation.cc:407] Connection header received
0924 04:17:34.704288 (+ 853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:34.704292 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:34.704348 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:34.704422 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:17:34.705538 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:34.706188 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:34.707367 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:34.707624 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:34.707712 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:34.709914 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:17:34.709917 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:17:34.709919 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:17:34.709941 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:17:34.711014 (+ 1073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:17:34.711483 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:17:34.711485 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:17:34.711486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:17:34.711553 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:17:34.711825 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:17:34.711826 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:17:34.711828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:17:34.711977 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:17:34.712042 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:34.712463 (+ 421us) server_negotiation.cc:299] Negotiation successful
0924 04:17:34.712553 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 04:17:34.713398 17913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:34.702874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36411 (local address 127.0.98.3:33039)
0924 04:17:34.703299 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:34.703339 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 04:17:34.704094 (+ 755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:17:34.704587 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:17:34.704594 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:17:34.704934 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:17:34.705407 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:34.705420 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:34.706303 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:34.706306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:34.707263 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:34.707273 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:34.707750 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:34.707754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:34.707912 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:34.708397 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:17:34.708412 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:17:34.709771 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:17:34.711114 (+ 1343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:17:34.711118 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:17:34.711129 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:17:34.711413 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:17:34.711619 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:17:34.711621 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:17:34.711622 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:17:34.711759 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:17:34.712046 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:17:34.712050 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:17:34.712283 (+ 233us) client_negotiation.cc:769] Sending connection context
0924 04:17:34.712421 (+ 138us) client_negotiation.cc:240] Negotiation successful
0924 04:17:34.712652 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":103,"threads_started":1}
I20260924 04:17:34.714339 17911 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36411
I20260924 04:17:34.714540 17911 heartbeater.cc:464] Registering TS with master...
I20260924 04:17:34.714967 17911 heartbeater.cc:511] Master 127.0.98.62:36411 requested a full tablet report, sending...
I20260924 04:17:34.715894 17452 ts_manager.cc:194] Registered new tserver with Master: 94591a7836b54cf3b71eb70e48842f9c (127.0.98.3:42569)
I20260924 04:17:34.716393 17452 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33039
I20260924 04:17:34.719260 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:17:34.728559 17491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:34.721026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54226 (local address 127.0.98.62:36411)
0924 04:17:34.721147 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:34.721151 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:34.721448 (+ 297us) server_negotiation.cc:407] Connection header received
0924 04:17:34.721564 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:34.721567 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:34.721639 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:34.721698 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:17:34.722336 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:34.722829 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:34.723671 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:34.723887 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:34.723967 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:34.725349 (+ 1382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:17:34.725353 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:17:34.725355 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:17:34.725380 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:17:34.726927 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:17:34.727361 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:17:34.727365 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:17:34.727366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:17:34.727455 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:17:34.727709 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:17:34.727713 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:17:34.727715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:17:34.727985 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:17:34.728108 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:34.728316 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 04:17:34.728414 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 04:17:34.731439 17452 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54226:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:17:34.734294 17452 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:17:34.747197 17925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:34.742264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42569 (local address 127.0.0.1:33240)
0924 04:17:34.742481 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:34.742493 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:17:34.742566 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:17:34.743214 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:17:34.743220 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:17:34.743275 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:17:34.743437 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:34.743444 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:34.744559 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:34.744562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:34.745779 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:34.745787 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:34.746690 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:34.746693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:34.746802 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:34.746840 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 04:17:34.746944 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 04:17:34.747050 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":70,"spinlock_wait_cycles":6400,"thread_start_us":87,"threads_started":1}
I20260924 04:17:34.747279 17926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:34.742451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33240 (local address 127.0.98.3:42569)
0924 04:17:34.742819 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:34.742829 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:34.742846 (+ 17us) server_negotiation.cc:407] Connection header received
0924 04:17:34.742903 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:34.742907 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:34.743040 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:34.743570 (+ 530us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:17:34.743715 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:34.744444 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:34.745901 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:34.746580 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:34.746681 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:34.746813 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:34.746982 (+ 169us) server_negotiation.cc:299] Negotiation successful
0924 04:17:34.747073 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":109,"threads_started":1}
I20260924 04:17:34.747557 17922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:34.741719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43351 (local address 127.0.0.1:48404)
0924 04:17:34.742080 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:34.742097 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:17:34.742235 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:17:34.743273 (+ 1038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:17:34.743276 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:17:34.743294 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:17:34.743427 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:34.743432 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:34.744994 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:34.744997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:34.746192 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:34.746202 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:34.747192 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:34.747196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:34.747308 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:34.747331 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 04:17:34.747382 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 04:17:34.747445 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":5,"spinlock_wait_cycles":11392,"thread_start_us":91,"threads_started":1}
I20260924 04:17:34.747802 17923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:34.741602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41835 (local address 127.0.0.1:54426)
0924 04:17:34.742051 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:34.742104 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 04:17:34.742235 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:17:34.743214 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:17:34.743220 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:17:34.743270 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:17:34.743601 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:34.743607 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:34.745419 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:34.745422 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:34.746447 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:34.746454 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:34.747458 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:34.747462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:34.747557 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:34.747579 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 04:17:34.747619 (+ 40us) client_negotiation.cc:240] Negotiation successful
0924 04:17:34.747679 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":211,"spinlock_wait_cycles":11264,"thread_start_us":74,"threads_started":1}
I20260924 04:17:34.748063 17927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:34.742494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48404 (local address 127.0.98.1:43351)
0924 04:17:34.742793 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:34.742804 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:34.742826 (+ 22us) server_negotiation.cc:407] Connection header received
0924 04:17:34.742897 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:34.742902 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:34.743107 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:34.743549 (+ 442us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:17:34.743678 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:34.744861 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:34.746315 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:34.747060 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:34.747645 (+ 585us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:34.747774 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:34.747824 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 04:17:34.747910 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":126,"threads_started":1}
I20260924 04:17:34.748364 17924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:34.741739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54426 (local address 127.0.98.2:41835)
0924 04:17:34.742731 (+ 992us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:34.742742 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:34.742767 (+ 25us) server_negotiation.cc:407] Connection header received
0924 04:17:34.742841 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:34.742846 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:34.743041 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:34.743363 (+ 322us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:17:34.743729 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:34.744870 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:34.746545 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:34.747335 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:34.747944 (+ 609us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:34.748085 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:34.748131 (+ 46us) server_negotiation.cc:299] Negotiation successful
0924 04:17:34.748226 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":870,"thread_start_us":97,"threads_started":1}
I20260924 04:17:34.750491 17577 tablet_service.cc:1467] Processing CreateTablet for tablet 749b48e1a8a041f08052c8bf0a533217 (DEFAULT_TABLE table=test-table [id=23525754f3744dea8dfbf093bec56e1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:17:34.750588 17711 tablet_service.cc:1467] Processing CreateTablet for tablet 749b48e1a8a041f08052c8bf0a533217 (DEFAULT_TABLE table=test-table [id=23525754f3744dea8dfbf093bec56e1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:17:34.750588 17845 tablet_service.cc:1467] Processing CreateTablet for tablet 749b48e1a8a041f08052c8bf0a533217 (DEFAULT_TABLE table=test-table [id=23525754f3744dea8dfbf093bec56e1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:17:34.751462 17577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 749b48e1a8a041f08052c8bf0a533217. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:17:34.751554 17711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 749b48e1a8a041f08052c8bf0a533217. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:17:34.751554 17845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 749b48e1a8a041f08052c8bf0a533217. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:17:34.757993 17928 tablet_bootstrap.cc:492] T 749b48e1a8a041f08052c8bf0a533217 P 13f78c4a044144c08a5fe644c6b998bb: Bootstrap starting.
I20260924 04:17:34.758769 17929 tablet_bootstrap.cc:492] T 749b48e1a8a041f08052c8bf0a533217 P f3bae83e22174b55a109bdacb492e434: Bootstrap starting.
I20260924 04:17:34.758767 17930 tablet_bootstrap.cc:492] T 749b48e1a8a041f08052c8bf0a533217 P 94591a7836b54cf3b71eb70e48842f9c: Bootstrap starting.
I20260924 04:17:34.760052 17928 tablet_bootstrap.cc:654] T 749b48e1a8a041f08052c8bf0a533217 P 13f78c4a044144c08a5fe644c6b998bb: Neither blocks nor log segments found. Creating new log.
I20260924 04:17:34.760910 17930 tablet_bootstrap.cc:654] T 749b48e1a8a041f08052c8bf0a533217 P 94591a7836b54cf3b71eb70e48842f9c: Neither blocks nor log segments found. Creating new log.
I20260924 04:17:34.760910 17929 tablet_bootstrap.cc:654] T 749b48e1a8a041f08052c8bf0a533217 P f3bae83e22174b55a109bdacb492e434: Neither blocks nor log segments found. Creating new log.
I20260924 04:17:34.761019 17928 log.cc:824] T 749b48e1a8a041f08052c8bf0a533217 P 13f78c4a044144c08a5fe644c6b998bb: Log is configured to *not* fsync() on all Append() calls
I20260924 04:17:34.762107 17930 log.cc:824] T 749b48e1a8a041f08052c8bf0a533217 P 94591a7836b54cf3b71eb70e48842f9c: Log is configured to *not* fsync() on all Append() calls
I20260924 04:17:34.762107 17929 log.cc:824] T 749b48e1a8a041f08052c8bf0a533217 P f3bae83e22174b55a109bdacb492e434: Log is configured to *not* fsync() on all Append() calls
I20260924 04:17:34.762926 17928 tablet_bootstrap.cc:492] T 749b48e1a8a041f08052c8bf0a533217 P 13f78c4a044144c08a5fe644c6b998bb: No bootstrap required, opened a new log
I20260924 04:17:34.763372 17928 ts_tablet_manager.cc:1397] T 749b48e1a8a041f08052c8bf0a533217 P 13f78c4a044144c08a5fe644c6b998bb: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:17:34.763945 17930 tablet_bootstrap.cc:492] T 749b48e1a8a041f08052c8bf0a533217 P 94591a7836b54cf3b71eb70e48842f9c: No bootstrap required, opened a new log
I20260924 04:17:34.763942 17929 tablet_bootstrap.cc:492] T 749b48e1a8a041f08052c8bf0a533217 P f3bae83e22174b55a109bdacb492e434: No bootstrap required, opened a new log
I20260924 04:17:34.764125 17930 ts_tablet_manager.cc:1397] T 749b48e1a8a041f08052c8bf0a533217 P 94591a7836b54cf3b71eb70e48842f9c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:17:34.764125 17929 ts_tablet_manager.cc:1397] T 749b48e1a8a041f08052c8bf0a533217 P f3bae83e22174b55a109bdacb492e434: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:17:34.766254 17928 raft_consensus.cc:348] T 749b48e1a8a041f08052c8bf0a533217 P 13f78c4a044144c08a5fe644c6b998bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bae83e22174b55a109bdacb492e434" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41835 } } peers { permanent_uuid: "13f78c4a044144c08a5fe644c6b998bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43351 } } peers { permanent_uuid: "94591a7836b54cf3b71eb70e48842f9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42569 } }
I20260924 04:17:34.766481 17928 raft_consensus.cc:374] T 749b48e1a8a041f08052c8bf0a533217 P 13f78c4a044144c08a5fe644c6b998bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:17:34.766557 17928 raft_consensus.cc:729] T 749b48e1a8a041f08052c8bf0a533217 P 13f78c4a044144c08a5fe644c6b998bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13f78c4a044144c08a5fe644c6b998bb, State: Initialized, Role: FOLLOWER
I20260924 04:17:34.767017 17928 consensus_queue.cc:260] T 749b48e1a8a041f08052c8bf0a533217 P 13f78c4a044144c08a5fe644c6b998bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bae83e22174b55a109bdacb492e434" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41835 } } peers { permanent_uuid: "13f78c4a044144c08a5fe644c6b998bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43351 } } peers { permanent_uuid: "94591a7836b54cf3b71eb70e48842f9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42569 } }
I20260924 04:17:34.767910 17928 ts_tablet_manager.cc:1428] T 749b48e1a8a041f08052c8bf0a533217 P 13f78c4a044144c08a5fe644c6b998bb: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260924 04:17:34.767931 17930 raft_consensus.cc:348] T 749b48e1a8a041f08052c8bf0a533217 P 94591a7836b54cf3b71eb70e48842f9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bae83e22174b55a109bdacb492e434" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41835 } } peers { permanent_uuid: "13f78c4a044144c08a5fe644c6b998bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43351 } } peers { permanent_uuid: "94591a7836b54cf3b71eb70e48842f9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42569 } }
I20260924 04:17:34.767931 17929 raft_consensus.cc:348] T 749b48e1a8a041f08052c8bf0a533217 P f3bae83e22174b55a109bdacb492e434 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bae83e22174b55a109bdacb492e434" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41835 } } peers { permanent_uuid: "13f78c4a044144c08a5fe644c6b998bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43351 } } peers { permanent_uuid: "94591a7836b54cf3b71eb70e48842f9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42569 } }
I20260924 04:17:34.768198 17929 raft_consensus.cc:374] T 749b48e1a8a041f08052c8bf0a533217 P f3bae83e22174b55a109bdacb492e434 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:17:34.768227 17930 raft_consensus.cc:374] T 749b48e1a8a041f08052c8bf0a533217 P 94591a7836b54cf3b71eb70e48842f9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:17:34.768235 17643 heartbeater.cc:503] Master 127.0.98.62:36411 was elected leader, sending a full tablet report...
I20260924 04:17:34.768275 17929 raft_consensus.cc:729] T 749b48e1a8a041f08052c8bf0a533217 P f3bae83e22174b55a109bdacb492e434 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3bae83e22174b55a109bdacb492e434, State: Initialized, Role: FOLLOWER
I20260924 04:17:34.768289 17930 raft_consensus.cc:729] T 749b48e1a8a041f08052c8bf0a533217 P 94591a7836b54cf3b71eb70e48842f9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94591a7836b54cf3b71eb70e48842f9c, State: Initialized, Role: FOLLOWER
I20260924 04:17:34.768720 17929 consensus_queue.cc:260] T 749b48e1a8a041f08052c8bf0a533217 P f3bae83e22174b55a109bdacb492e434 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bae83e22174b55a109bdacb492e434" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41835 } } peers { permanent_uuid: "13f78c4a044144c08a5fe644c6b998bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43351 } } peers { permanent_uuid: "94591a7836b54cf3b71eb70e48842f9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42569 } }
I20260924 04:17:34.768720 17930 consensus_queue.cc:260] T 749b48e1a8a041f08052c8bf0a533217 P 94591a7836b54cf3b71eb70e48842f9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bae83e22174b55a109bdacb492e434" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41835 } } peers { permanent_uuid: "13f78c4a044144c08a5fe644c6b998bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43351 } } peers { permanent_uuid: "94591a7836b54cf3b71eb70e48842f9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42569 } }
I20260924 04:17:34.769333 17777 heartbeater.cc:503] Master 127.0.98.62:36411 was elected leader, sending a full tablet report...
I20260924 04:17:34.769713 17930 ts_tablet_manager.cc:1428] T 749b48e1a8a041f08052c8bf0a533217 P 94591a7836b54cf3b71eb70e48842f9c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 04:17:34.770004 17911 heartbeater.cc:503] Master 127.0.98.62:36411 was elected leader, sending a full tablet report...
I20260924 04:17:34.771097 17929 ts_tablet_manager.cc:1428] T 749b48e1a8a041f08052c8bf0a533217 P f3bae83e22174b55a109bdacb492e434: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260924 04:17:34.794281 17644 tablet.cc:2369] T 749b48e1a8a041f08052c8bf0a533217 P 13f78c4a044144c08a5fe644c6b998bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:17:34.873567 17936 raft_consensus.cc:482] T 749b48e1a8a041f08052c8bf0a533217 P 94591a7836b54cf3b71eb70e48842f9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:17:34.873768 17936 raft_consensus.cc:504] T 749b48e1a8a041f08052c8bf0a533217 P 94591a7836b54cf3b71eb70e48842f9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bae83e22174b55a109bdacb492e434" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41835 } } peers { permanent_uuid: "13f78c4a044144c08a5fe644c6b998bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43351 } } peers { permanent_uuid: "94591a7836b54cf3b71eb70e48842f9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42569 } }
I20260924 04:17:34.874886 17936 leader_election.cc:288] T 749b48e1a8a041f08052c8bf0a533217 P 94591a7836b54cf3b71eb70e48842f9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3bae83e22174b55a109bdacb492e434 (127.0.98.2:41835), 13f78c4a044144c08a5fe644c6b998bb (127.0.98.1:43351)
I20260924 04:17:34.879611 17927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:34.875346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49897 (local address 127.0.98.1:43351)
0924 04:17:34.875468 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:34.875473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:34.875554 (+ 81us) server_negotiation.cc:407] Connection header received
0924 04:17:34.875632 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:34.875634 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:34.875676 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:34.875743 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:17:34.875960 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:34.876726 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:34.878415 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:34.879022 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:34.879134 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:34.879198 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:34.879412 (+ 214us) server_negotiation.cc:299] Negotiation successful
0924 04:17:34.879481 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:17:34.879617 17924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:34.875171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37133 (local address 127.0.98.2:41835)
0924 04:17:34.875312 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:34.875316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:34.875330 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:17:34.875504 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:34.875508 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:34.875557 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:34.875635 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:17:34.875899 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:34.876716 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:34.878415 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:34.879059 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:34.879147 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:34.879198 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:34.879416 (+ 218us) server_negotiation.cc:299] Negotiation successful
0924 04:17:34.879481 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 04:17:34.879664 17913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:34.875075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41835 (local address 127.0.98.3:37133)
0924 04:17:34.875275 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:34.875289 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:17:34.875379 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:17:34.875666 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:17:34.875668 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:17:34.875681 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:17:34.875795 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:34.875801 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:34.876837 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:34.876841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:34.878279 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:34.878290 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:34.879147 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:34.879150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:34.879314 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:34.879331 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 04:17:34.879385 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 04:17:34.879501 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":177}
I20260924 04:17:34.879671 17937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:34.875156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43351 (local address 127.0.98.3:49897)
0924 04:17:34.875478 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:34.875490 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:17:34.875560 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:17:34.875760 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:17:34.875763 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:17:34.875776 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:17:34.875869 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:34.875876 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:34.876837 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:34.876841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:34.878284 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:34.878290 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:34.879132 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:34.879136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:34.879288 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:34.879314 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 04:17:34.879391 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 04:17:34.879501 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":137,"thread_start_us":129,"threads_started":1}
I20260924 04:17:34.880345 17597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "749b48e1a8a041f08052c8bf0a533217" candidate_uuid: "94591a7836b54cf3b71eb70e48842f9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13f78c4a044144c08a5fe644c6b998bb" is_pre_election: true
I20260924 04:17:34.880348 17731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "749b48e1a8a041f08052c8bf0a533217" candidate_uuid: "94591a7836b54cf3b71eb70e48842f9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3bae83e22174b55a109bdacb492e434" is_pre_election: true
I20260924 04:17:34.880658 17597 raft_consensus.cc:2454] T 749b48e1a8a041f08052c8bf0a533217 P 13f78c4a044144c08a5fe644c6b998bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94591a7836b54cf3b71eb70e48842f9c in term 0.
I20260924 04:17:34.880658 17731 raft_consensus.cc:2454] T 749b48e1a8a041f08052c8bf0a533217 P f3bae83e22174b55a109bdacb492e434 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94591a7836b54cf3b71eb70e48842f9c in term 0.
I20260924 04:17:34.881119 17798 leader_election.cc:302] T 749b48e1a8a041f08052c8bf0a533217 P 94591a7836b54cf3b71eb70e48842f9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94591a7836b54cf3b71eb70e48842f9c, f3bae83e22174b55a109bdacb492e434; no voters:
I20260924 04:17:34.881364 17936 raft_consensus.cc:2781] T 749b48e1a8a041f08052c8bf0a533217 P 94591a7836b54cf3b71eb70e48842f9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:17:34.881439 17936 raft_consensus.cc:482] T 749b48e1a8a041f08052c8bf0a533217 P 94591a7836b54cf3b71eb70e48842f9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:17:34.881485 17936 raft_consensus.cc:3037] T 749b48e1a8a041f08052c8bf0a533217 P 94591a7836b54cf3b71eb70e48842f9c [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:17:34.882438 17936 raft_consensus.cc:504] T 749b48e1a8a041f08052c8bf0a533217 P 94591a7836b54cf3b71eb70e48842f9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bae83e22174b55a109bdacb492e434" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41835 } } peers { permanent_uuid: "13f78c4a044144c08a5fe644c6b998bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43351 } } peers { permanent_uuid: "94591a7836b54cf3b71eb70e48842f9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42569 } }
I20260924 04:17:34.882752 17936 leader_election.cc:288] T 749b48e1a8a041f08052c8bf0a533217 P 94591a7836b54cf3b71eb70e48842f9c [CANDIDATE]: Term 1 election: Requested vote from peers f3bae83e22174b55a109bdacb492e434 (127.0.98.2:41835), 13f78c4a044144c08a5fe644c6b998bb (127.0.98.1:43351)
I20260924 04:17:34.883037 17597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "749b48e1a8a041f08052c8bf0a533217" candidate_uuid: "94591a7836b54cf3b71eb70e48842f9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13f78c4a044144c08a5fe644c6b998bb"
I20260924 04:17:34.883073 17731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "749b48e1a8a041f08052c8bf0a533217" candidate_uuid: "94591a7836b54cf3b71eb70e48842f9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3bae83e22174b55a109bdacb492e434"
I20260924 04:17:34.883142 17597 raft_consensus.cc:3037] T 749b48e1a8a041f08052c8bf0a533217 P 13f78c4a044144c08a5fe644c6b998bb [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:17:34.883180 17731 raft_consensus.cc:3037] T 749b48e1a8a041f08052c8bf0a533217 P f3bae83e22174b55a109bdacb492e434 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:17:34.884145 17597 raft_consensus.cc:2454] T 749b48e1a8a041f08052c8bf0a533217 P 13f78c4a044144c08a5fe644c6b998bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94591a7836b54cf3b71eb70e48842f9c in term 1.
I20260924 04:17:34.884162 17731 raft_consensus.cc:2454] T 749b48e1a8a041f08052c8bf0a533217 P f3bae83e22174b55a109bdacb492e434 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94591a7836b54cf3b71eb70e48842f9c in term 1.
I20260924 04:17:34.884418 17801 leader_election.cc:302] T 749b48e1a8a041f08052c8bf0a533217 P 94591a7836b54cf3b71eb70e48842f9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13f78c4a044144c08a5fe644c6b998bb, 94591a7836b54cf3b71eb70e48842f9c; no voters:
I20260924 04:17:34.884632 17936 raft_consensus.cc:2781] T 749b48e1a8a041f08052c8bf0a533217 P 94591a7836b54cf3b71eb70e48842f9c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:17:34.884826 17936 raft_consensus.cc:686] T 749b48e1a8a041f08052c8bf0a533217 P 94591a7836b54cf3b71eb70e48842f9c [term 1 LEADER]: Becoming Leader. State: Replica: 94591a7836b54cf3b71eb70e48842f9c, State: Running, Role: LEADER
I20260924 04:17:34.885118 17936 consensus_queue.cc:237] T 749b48e1a8a041f08052c8bf0a533217 P 94591a7836b54cf3b71eb70e48842f9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bae83e22174b55a109bdacb492e434" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41835 } } peers { permanent_uuid: "13f78c4a044144c08a5fe644c6b998bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43351 } } peers { permanent_uuid: "94591a7836b54cf3b71eb70e48842f9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42569 } }
I20260924 04:17:34.887939 17452 catalog_manager.cc:5495] T 749b48e1a8a041f08052c8bf0a533217 P 94591a7836b54cf3b71eb70e48842f9c reported cstate change: term changed from 0 to 1, leader changed from <none> to 94591a7836b54cf3b71eb70e48842f9c (127.0.98.3). New cstate: current_term: 1 leader_uuid: "94591a7836b54cf3b71eb70e48842f9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bae83e22174b55a109bdacb492e434" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13f78c4a044144c08a5fe644c6b998bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94591a7836b54cf3b71eb70e48842f9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42569 } health_report { overall_health: HEALTHY } } }
I20260924 04:17:34.897578 17926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:34.894583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33252 (local address 127.0.98.3:42569)
0924 04:17:34.894690 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:34.894693 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:34.894705 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:17:34.894825 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:34.894827 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:34.894874 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:34.894945 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:17:34.895120 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:34.895635 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:34.896433 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:34.896628 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:34.896708 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:34.896905 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:17:34.897275 (+ 370us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:17:34.897356 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:34.897397 (+ 41us) server_negotiation.cc:299] Negotiation successful
0924 04:17:34.897438 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 04:17:34.902916 17597 raft_consensus.cc:1261] T 749b48e1a8a041f08052c8bf0a533217 P 13f78c4a044144c08a5fe644c6b998bb [term 1 FOLLOWER]: Refusing update from remote peer 94591a7836b54cf3b71eb70e48842f9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:17:34.902916 17731 raft_consensus.cc:1261] T 749b48e1a8a041f08052c8bf0a533217 P f3bae83e22174b55a109bdacb492e434 [term 1 FOLLOWER]: Refusing update from remote peer 94591a7836b54cf3b71eb70e48842f9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:17:34.903573 17938 consensus_queue.cc:1035] T 749b48e1a8a041f08052c8bf0a533217 P 94591a7836b54cf3b71eb70e48842f9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3bae83e22174b55a109bdacb492e434" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:17:34.903731 17936 consensus_queue.cc:1035] T 749b48e1a8a041f08052c8bf0a533217 P 94591a7836b54cf3b71eb70e48842f9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "13f78c4a044144c08a5fe644c6b998bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:17:34.910624 17942 mvcc.cc:205] Tried to move back new op lower bound from 7332755271275728896 to 7332755271211839488. Current Snapshot: MvccSnapshot[applied={T|T < 7332755271275728896}]
I20260924 04:17:34.913036 17943 mvcc.cc:205] Tried to move back new op lower bound from 7332755271275728896 to 7332755271211839488. Current Snapshot: MvccSnapshot[applied={T|T < 7332755271275728896}]
I20260924 04:17:34.918210 17452 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54226:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:17:34.918397 17452 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54226:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:17:34.920692 17452 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2f35f35222b3419eaf87d8ecce217401: Sending DeleteTablet for 3 replicas of tablet 749b48e1a8a041f08052c8bf0a533217
I20260924 04:17:34.921458 17577 tablet_service.cc:1514] Processing DeleteTablet for tablet 749b48e1a8a041f08052c8bf0a533217 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:17:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48404
I20260924 04:17:34.921540 17845 tablet_service.cc:1514] Processing DeleteTablet for tablet 749b48e1a8a041f08052c8bf0a533217 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:17:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33240
I20260924 04:17:34.921525 17711 tablet_service.cc:1514] Processing DeleteTablet for tablet 749b48e1a8a041f08052c8bf0a533217 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:17:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54426
I20260924 04:17:34.921939 17953 tablet_replica.cc:330] T 749b48e1a8a041f08052c8bf0a533217 P f3bae83e22174b55a109bdacb492e434: stopping tablet replica
I20260924 04:17:34.922081 17951 tablet_replica.cc:330] T 749b48e1a8a041f08052c8bf0a533217 P 13f78c4a044144c08a5fe644c6b998bb: stopping tablet replica
I20260924 04:17:34.922286 17953 raft_consensus.cc:2229] T 749b48e1a8a041f08052c8bf0a533217 P f3bae83e22174b55a109bdacb492e434 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:17:34.922334 17951 raft_consensus.cc:2229] T 749b48e1a8a041f08052c8bf0a533217 P 13f78c4a044144c08a5fe644c6b998bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:17:34.922652 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 17514
I20260924 04:17:34.923216 17953 raft_consensus.cc:2258] T 749b48e1a8a041f08052c8bf0a533217 P f3bae83e22174b55a109bdacb492e434 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:17:34.923568 17952 tablet_replica.cc:330] T 749b48e1a8a041f08052c8bf0a533217 P 94591a7836b54cf3b71eb70e48842f9c: stopping tablet replica
I20260924 04:17:34.923811 17952 raft_consensus.cc:2229] T 749b48e1a8a041f08052c8bf0a533217 P 94591a7836b54cf3b71eb70e48842f9c [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:17:34.924100 17952 raft_consensus.cc:2258] T 749b48e1a8a041f08052c8bf0a533217 P 94591a7836b54cf3b71eb70e48842f9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:17:34.925333 17952 ts_tablet_manager.cc:1905] T 749b48e1a8a041f08052c8bf0a533217 P 94591a7836b54cf3b71eb70e48842f9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:17:34.925870 17953 ts_tablet_manager.cc:1905] T 749b48e1a8a041f08052c8bf0a533217 P f3bae83e22174b55a109bdacb492e434: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:17:34.927860 17952 ts_tablet_manager.cc:1918] T 749b48e1a8a041f08052c8bf0a533217 P 94591a7836b54cf3b71eb70e48842f9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:17:34.927947 17952 log.cc:1196] T 749b48e1a8a041f08052c8bf0a533217 P 94591a7836b54cf3b71eb70e48842f9c: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/ts-2/wal/wals/749b48e1a8a041f08052c8bf0a533217
I20260924 04:17:34.927973 17953 ts_tablet_manager.cc:1918] T 749b48e1a8a041f08052c8bf0a533217 P f3bae83e22174b55a109bdacb492e434: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:17:34.928108 17953 log.cc:1196] T 749b48e1a8a041f08052c8bf0a533217 P f3bae83e22174b55a109bdacb492e434: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790223216265354-392-0/minicluster-data/ts-1/wal/wals/749b48e1a8a041f08052c8bf0a533217
I20260924 04:17:34.928226 17952 ts_tablet_manager.cc:1939] T 749b48e1a8a041f08052c8bf0a533217 P 94591a7836b54cf3b71eb70e48842f9c: Deleting consensus metadata
I20260924 04:17:34.928429 17953 ts_tablet_manager.cc:1939] T 749b48e1a8a041f08052c8bf0a533217 P f3bae83e22174b55a109bdacb492e434: Deleting consensus metadata
W20260924 04:17:34.928943 17439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43351 (error 108)
I20260924 04:17:34.929580 17436 catalog_manager.cc:4841] TS f3bae83e22174b55a109bdacb492e434 (127.0.98.2:41835): tablet 749b48e1a8a041f08052c8bf0a533217 (table test-table [id=23525754f3744dea8dfbf093bec56e1e]) successfully deleted
I20260924 04:17:34.929768 17436 catalog_manager.cc:4841] TS 94591a7836b54cf3b71eb70e48842f9c (127.0.98.3:42569): tablet 749b48e1a8a041f08052c8bf0a533217 (table test-table [id=23525754f3744dea8dfbf093bec56e1e]) successfully deleted
I20260924 04:17:34.929867 17925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:34.929602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43351 (local address 127.0.0.1:48408)
0924 04:17:34.929719 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:34.929787 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 04:17:34.930054 17439 catalog_manager.cc:4584] TS 13f78c4a044144c08a5fe644c6b998bb (127.0.98.1:43351): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 749b48e1a8a041f08052c8bf0a533217: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43351: connect: Connection refused (error 111)
I20260924 04:17:34.930174 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 17648
I20260924 04:17:34.935360 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 17782
I20260924 04:17:34.939905 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 17420
2026-09-24T04:17:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2153 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-24T04:17:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:17:34Z Disabled control of system clock
I20260924 04:17:34.967060 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43427
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43427
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 04:17:35.074339 17958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:17:35.074589 17958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:17:35.074623 17958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:17:35.078612 17958 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:17:35.078681 17958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:17:35.078696 17958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:17:35.078711 17958 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:17:35.078724 17958 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:17:35.083577 17958 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43427
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:17:35.084607 17958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:17:35.085933 17958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:17:35.094725 17964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:17:35.094765 17963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:17:35.095146 17966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:17:35.095490 17958 server_base.cc:1034] running on GCE node
I20260924 04:17:35.096227 17958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:17:35.097671 17958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:17:35.098817 17958 hybrid_clock.cc:648] HybridClock initialized: now 1790223455098798 us; error 40 us; skew 500 ppm
I20260924 04:17:35.100997 17958 webserver.cc:460] Webserver started at http://127.0.98.62:38731/ using document root <none> and password file <none>
I20260924 04:17:35.101650 17958 fs_manager.cc:359] Metadata directory not provided
I20260924 04:17:35.101742 17958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:17:35.102084 17958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:17:35.104887 17958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/master-0/data/instance:
uuid: "2f504f029beb41c38e474516a151de95"
format_stamp: "Formatted at 2026-09-24 04:17:35 on dist-test-slave-p401"
server_key: "193529a17d900592149283bd78208aae"
server_key_iv: "97e29276dfed78c578b1777a68cd6799"
server_key_version: "encryptionkey@0"
I20260924 04:17:35.105572 17958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/master-0/wal/instance:
uuid: "2f504f029beb41c38e474516a151de95"
format_stamp: "Formatted at 2026-09-24 04:17:35 on dist-test-slave-p401"
server_key: "193529a17d900592149283bd78208aae"
server_key_iv: "97e29276dfed78c578b1777a68cd6799"
server_key_version: "encryptionkey@0"
I20260924 04:17:35.110097 17958 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 04:17:35.113236 17972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:35.114758 17958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:17:35.114879 17958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/master-0/wal
uuid: "2f504f029beb41c38e474516a151de95"
format_stamp: "Formatted at 2026-09-24 04:17:35 on dist-test-slave-p401"
server_key: "193529a17d900592149283bd78208aae"
server_key_iv: "97e29276dfed78c578b1777a68cd6799"
server_key_version: "encryptionkey@0"
I20260924 04:17:35.115003 17958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:17:35.136490 17958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:17:35.137168 17958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:17:35.137333 17958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:17:35.143961 17958 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43427
I20260924 04:17:35.143980 18024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43427 every 8 connection(s)
I20260924 04:17:35.144918 17958 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:17:35.147945 18025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:17:35.152736 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 17958
I20260924 04:17:35.152854 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/master-0/wal/instance
I20260924 04:17:35.153141 392 external_mini_cluster.cc:1428] Setting key 331f038b57ba2fb83eb8a997520aa084
I20260924 04:17:35.153568 18025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f504f029beb41c38e474516a151de95: Bootstrap starting.
I20260924 04:17:35.156133 18025 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f504f029beb41c38e474516a151de95: Neither blocks nor log segments found. Creating new log.
I20260924 04:17:35.156842 18025 log.cc:824] T 00000000000000000000000000000000 P 2f504f029beb41c38e474516a151de95: Log is configured to *not* fsync() on all Append() calls
I20260924 04:17:35.158710 18025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f504f029beb41c38e474516a151de95: No bootstrap required, opened a new log
I20260924 04:17:35.162456 18028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:35.154538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37624 (local address 127.0.98.62:43427)
0924 04:17:35.155443 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:35.155462 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:35.155507 (+ 45us) server_negotiation.cc:407] Connection header received
0924 04:17:35.156148 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:35.156164 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:35.156586 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:35.156890 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:17:35.157311 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:35.158268 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:35.159260 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:35.159738 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:35.159888 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:35.160755 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:17:35.160760 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:17:35.160773 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:17:35.161162 (+ 389us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:17:35.161222 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:17:35.161242 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:17:35.161323 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:35.161464 (+ 141us) server_negotiation.cc:299] Negotiation successful
0924 04:17:35.161717 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":132,"threads_started":1}
I20260924 04:17:35.162372 18025 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f504f029beb41c38e474516a151de95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f504f029beb41c38e474516a151de95" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43427 } }
I20260924 04:17:35.162601 18025 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f504f029beb41c38e474516a151de95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:17:35.162678 18025 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f504f029beb41c38e474516a151de95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f504f029beb41c38e474516a151de95, State: Initialized, Role: FOLLOWER
I20260924 04:17:35.163067 18025 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f504f029beb41c38e474516a151de95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f504f029beb41c38e474516a151de95" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43427 } }
I20260924 04:17:35.163193 18025 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f504f029beb41c38e474516a151de95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:17:35.163273 18025 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f504f029beb41c38e474516a151de95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:17:35.163369 18025 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f504f029beb41c38e474516a151de95 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:17:35.164247 18025 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f504f029beb41c38e474516a151de95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f504f029beb41c38e474516a151de95" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43427 } }
I20260924 04:17:35.164558 18025 leader_election.cc:302] T 00000000000000000000000000000000 P 2f504f029beb41c38e474516a151de95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f504f029beb41c38e474516a151de95; no voters:
I20260924 04:17:35.164849 18025 leader_election.cc:288] T 00000000000000000000000000000000 P 2f504f029beb41c38e474516a151de95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:17:35.164937 18030 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f504f029beb41c38e474516a151de95 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:17:35.165177 18030 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f504f029beb41c38e474516a151de95 [term 1 LEADER]: Becoming Leader. State: Replica: 2f504f029beb41c38e474516a151de95, State: Running, Role: LEADER
I20260924 04:17:35.165457 18030 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f504f029beb41c38e474516a151de95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f504f029beb41c38e474516a151de95" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43427 } }
I20260924 04:17:35.166011 18025 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f504f029beb41c38e474516a151de95 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:17:35.167071 18032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f504f029beb41c38e474516a151de95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f504f029beb41c38e474516a151de95. Latest consensus state: current_term: 1 leader_uuid: "2f504f029beb41c38e474516a151de95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f504f029beb41c38e474516a151de95" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43427 } } }
I20260924 04:17:35.167073 18031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f504f029beb41c38e474516a151de95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f504f029beb41c38e474516a151de95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f504f029beb41c38e474516a151de95" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43427 } } }
I20260924 04:17:35.167217 18031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f504f029beb41c38e474516a151de95 [sys.catalog]: This master's current role is: LEADER
I20260924 04:17:35.167217 18032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f504f029beb41c38e474516a151de95 [sys.catalog]: This master's current role is: LEADER
I20260924 04:17:35.167654 18039 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:17:35.170670 18039 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:17:35.175864 18039 catalog_manager.cc:1347] Generated new cluster ID: 137cc89ac4454ef18b85e3f20fa4ba59
I20260924 04:17:35.175937 18039 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:17:35.193279 18039 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:17:35.194252 18039 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:17:35.203254 18039 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f504f029beb41c38e474516a151de95: Generated new TSK 0
I20260924 04:17:35.203964 18039 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:17:35.211418 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43427
--builtin_ntp_servers=127.0.98.20:33221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 04:17:35.323999 18049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:17:35.324249 18049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:17:35.324303 18049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:17:35.328243 18049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:17:35.328293 18049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:17:35.328366 18049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:17:35.333062 18049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18049
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:17:35.334033 18049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:17:35.335359 18049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:17:35.344305 18055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:17:35.344372 18054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:17:35.344305 18057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:17:35.344409 18049 server_base.cc:1034] running on GCE node
I20260924 04:17:35.344872 18049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:17:35.345487 18049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:17:35.346660 18049 hybrid_clock.cc:648] HybridClock initialized: now 1790223455346644 us; error 32 us; skew 500 ppm
I20260924 04:17:35.348513 18049 webserver.cc:460] Webserver started at http://127.0.98.1:34873/ using document root <none> and password file <none>
I20260924 04:17:35.349050 18049 fs_manager.cc:359] Metadata directory not provided
I20260924 04:17:35.349121 18049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:17:35.349314 18049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:17:35.351374 18049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/ts-0/data/instance:
uuid: "5c6ec328147f44acb5ac4e4ce0aa656f"
format_stamp: "Formatted at 2026-09-24 04:17:35 on dist-test-slave-p401"
server_key: "525de22feaccf6795d1ddf3dc32a6297"
server_key_iv: "c55b1d510db8476a622fbb2d6199c643"
server_key_version: "encryptionkey@0"
I20260924 04:17:35.351961 18049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5c6ec328147f44acb5ac4e4ce0aa656f"
format_stamp: "Formatted at 2026-09-24 04:17:35 on dist-test-slave-p401"
server_key: "525de22feaccf6795d1ddf3dc32a6297"
server_key_iv: "c55b1d510db8476a622fbb2d6199c643"
server_key_version: "encryptionkey@0"
I20260924 04:17:35.355669 18049 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:17:35.358286 18063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:35.359572 18049 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:17:35.359694 18049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/ts-0/wal
uuid: "5c6ec328147f44acb5ac4e4ce0aa656f"
format_stamp: "Formatted at 2026-09-24 04:17:35 on dist-test-slave-p401"
server_key: "525de22feaccf6795d1ddf3dc32a6297"
server_key_iv: "c55b1d510db8476a622fbb2d6199c643"
server_key_version: "encryptionkey@0"
I20260924 04:17:35.359797 18049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:17:35.375787 18049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:17:35.376492 18049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:17:35.376674 18049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:17:35.377262 18049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:17:35.378214 18049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:17:35.378261 18049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:35.378314 18049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:17:35.378343 18049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:35.387001 18049 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37273
I20260924 04:17:35.387024 18176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37273 every 8 connection(s)
I20260924 04:17:35.387954 18049 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:17:35.397140 18028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:35.389893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53381 (local address 127.0.98.62:43427)
0924 04:17:35.390009 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:35.390013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:35.391072 (+ 1059us) server_negotiation.cc:407] Connection header received
0924 04:17:35.392507 (+ 1435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:35.392510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:35.392567 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:35.392639 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:17:35.393543 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:35.394080 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:35.395246 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:35.395497 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:35.395591 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:35.396649 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:17:35.396653 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:17:35.396656 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:17:35.396676 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:17:35.396697 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:17:35.396705 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:17:35.396775 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:35.396922 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 04:17:35.396988 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 04:17:35.398031 18179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:35.390279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43427 (local address 127.0.98.1:53381)
0924 04:17:35.390919 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:35.390960 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 04:17:35.392290 (+ 1330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:17:35.392822 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:17:35.392830 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:17:35.392917 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:17:35.393405 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:35.393417 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:35.394205 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:35.394208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:35.395091 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:35.395100 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:35.395643 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:35.395647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:35.395811 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:35.396383 (+ 572us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:17:35.396384 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:17:35.396427 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:17:35.396496 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:17:35.396785 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:17:35.396794 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:17:35.396800 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 04:17:35.396909 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 04:17:35.397218 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":116,"threads_started":1}
I20260924 04:17:35.398265 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 18049
I20260924 04:17:35.398383 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:17:35.398691 392 external_mini_cluster.cc:1428] Setting key 7877c805c0e6dc537737f517e90048bd
I20260924 04:17:35.399382 18177 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43427
I20260924 04:17:35.399622 18177 heartbeater.cc:464] Registering TS with master...
I20260924 04:17:35.400182 18177 heartbeater.cc:511] Master 127.0.98.62:43427 requested a full tablet report, sending...
I20260924 04:17:35.400632 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43427
--builtin_ntp_servers=127.0.98.20:33221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 04:17:35.401610 17989 ts_manager.cc:194] Registered new tserver with Master: 5c6ec328147f44acb5ac4e4ce0aa656f (127.0.98.1:37273)
I20260924 04:17:35.402730 17989 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:53381
W20260924 04:17:35.508565 18180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:17:35.508842 18180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:17:35.508881 18180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:17:35.512861 18180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:17:35.512912 18180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:17:35.512964 18180 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:17:35.517583 18180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:17:35.518553 18180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:17:35.519932 18180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:17:35.529206 18185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:17:35.529273 18186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:17:35.529327 18180 server_base.cc:1034] running on GCE node
W20260924 04:17:35.529275 18188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:17:35.529795 18180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:17:35.530402 18180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:17:35.531574 18180 hybrid_clock.cc:648] HybridClock initialized: now 1790223455531553 us; error 35 us; skew 500 ppm
I20260924 04:17:35.533324 18180 webserver.cc:460] Webserver started at http://127.0.98.2:39631/ using document root <none> and password file <none>
I20260924 04:17:35.533872 18180 fs_manager.cc:359] Metadata directory not provided
I20260924 04:17:35.533943 18180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:17:35.534144 18180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:17:35.536156 18180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/ts-1/data/instance:
uuid: "781be121ea5e4b4f9b8b3340b2d82958"
format_stamp: "Formatted at 2026-09-24 04:17:35 on dist-test-slave-p401"
server_key: "b415355b29426ae0214649ad54476a12"
server_key_iv: "b8016d76e7eaa48200488b6a3d3ee463"
server_key_version: "encryptionkey@0"
I20260924 04:17:35.536708 18180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/ts-1/wal/instance:
uuid: "781be121ea5e4b4f9b8b3340b2d82958"
format_stamp: "Formatted at 2026-09-24 04:17:35 on dist-test-slave-p401"
server_key: "b415355b29426ae0214649ad54476a12"
server_key_iv: "b8016d76e7eaa48200488b6a3d3ee463"
server_key_version: "encryptionkey@0"
I20260924 04:17:35.540236 18180 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:17:35.542563 18194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:35.543769 18180 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:17:35.543887 18180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/ts-1/wal
uuid: "781be121ea5e4b4f9b8b3340b2d82958"
format_stamp: "Formatted at 2026-09-24 04:17:35 on dist-test-slave-p401"
server_key: "b415355b29426ae0214649ad54476a12"
server_key_iv: "b8016d76e7eaa48200488b6a3d3ee463"
server_key_version: "encryptionkey@0"
I20260924 04:17:35.543982 18180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:17:35.558895 18180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:17:35.559594 18180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:17:35.559759 18180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:17:35.560314 18180 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:17:35.561213 18180 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:17:35.561259 18180 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:35.561311 18180 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:17:35.561337 18180 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:35.569440 18180 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43865
I20260924 04:17:35.569473 18307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43865 every 8 connection(s)
I20260924 04:17:35.570304 18180 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:17:35.576411 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 18180
I20260924 04:17:35.576529 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:17:35.576795 392 external_mini_cluster.cc:1428] Setting key 9e3f1f71036840ca0b6c63877e6d4038
I20260924 04:17:35.578890 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43427
--builtin_ntp_servers=127.0.98.20:33221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 04:17:35.579689 18028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:35.572105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37341 (local address 127.0.98.62:43427)
0924 04:17:35.572272 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:35.572277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:35.573319 (+ 1042us) server_negotiation.cc:407] Connection header received
0924 04:17:35.574609 (+ 1290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:35.574612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:35.574659 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:35.574719 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:17:35.575690 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:35.576258 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:35.577521 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:35.577856 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:35.577957 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:35.579210 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:17:35.579214 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:17:35.579216 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:17:35.579256 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:17:35.579281 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:17:35.579292 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:17:35.579364 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:35.579486 (+ 122us) server_negotiation.cc:299] Negotiation successful
0924 04:17:35.579542 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":68}
I20260924 04:17:35.580667 18310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:35.572461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43427 (local address 127.0.98.2:37341)
0924 04:17:35.573171 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:35.573213 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 04:17:35.574416 (+ 1203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:17:35.574878 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:17:35.574884 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:17:35.574968 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:17:35.575553 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:35.575565 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:35.576388 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:35.576391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:35.577393 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:35.577402 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:35.578072 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:35.578078 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:35.578260 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:35.578915 (+ 655us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:17:35.578917 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:17:35.578960 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:17:35.579035 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:17:35.579366 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:17:35.579371 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:17:35.579376 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 04:17:35.579470 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 04:17:35.579696 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":113,"threads_started":1}
I20260924 04:17:35.581863 18308 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43427
I20260924 04:17:35.582126 18308 heartbeater.cc:464] Registering TS with master...
I20260924 04:17:35.582737 18308 heartbeater.cc:511] Master 127.0.98.62:43427 requested a full tablet report, sending...
I20260924 04:17:35.583832 17989 ts_manager.cc:194] Registered new tserver with Master: 781be121ea5e4b4f9b8b3340b2d82958 (127.0.98.2:43865)
I20260924 04:17:35.584410 17989 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:37341
W20260924 04:17:35.685539 18311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:17:35.685781 18311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:17:35.685819 18311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:17:35.689745 18311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:17:35.689798 18311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:17:35.689863 18311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:17:35.694600 18311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:17:35.695623 18311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:17:35.696897 18311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:17:35.705850 18319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:17:35.705933 18311 server_base.cc:1034] running on GCE node
W20260924 04:17:35.705864 18316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:17:35.705863 18317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:17:35.706488 18311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:17:35.707072 18311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:17:35.708231 18311 hybrid_clock.cc:648] HybridClock initialized: now 1790223455708210 us; error 57 us; skew 500 ppm
I20260924 04:17:35.709944 18311 webserver.cc:460] Webserver started at http://127.0.98.3:43511/ using document root <none> and password file <none>
I20260924 04:17:35.710523 18311 fs_manager.cc:359] Metadata directory not provided
I20260924 04:17:35.710592 18311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:17:35.710788 18311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:17:35.712828 18311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/ts-2/data/instance:
uuid: "8cddcc2c27af45eb8bd7db873a1059f7"
format_stamp: "Formatted at 2026-09-24 04:17:35 on dist-test-slave-p401"
server_key: "d4a814dd6f003ca7529cf7aa0eee5701"
server_key_iv: "ff1812804e312eb289e222694db5d66a"
server_key_version: "encryptionkey@0"
I20260924 04:17:35.713353 18311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8cddcc2c27af45eb8bd7db873a1059f7"
format_stamp: "Formatted at 2026-09-24 04:17:35 on dist-test-slave-p401"
server_key: "d4a814dd6f003ca7529cf7aa0eee5701"
server_key_iv: "ff1812804e312eb289e222694db5d66a"
server_key_version: "encryptionkey@0"
I20260924 04:17:35.716858 18311 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:17:35.719132 18325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:35.720387 18311 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:17:35.720490 18311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/ts-2/wal
uuid: "8cddcc2c27af45eb8bd7db873a1059f7"
format_stamp: "Formatted at 2026-09-24 04:17:35 on dist-test-slave-p401"
server_key: "d4a814dd6f003ca7529cf7aa0eee5701"
server_key_iv: "ff1812804e312eb289e222694db5d66a"
server_key_version: "encryptionkey@0"
I20260924 04:17:35.720584 18311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:17:35.732554 18311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:17:35.733140 18311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:17:35.733287 18311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:17:35.733791 18311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:17:35.734635 18311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:17:35.734679 18311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:35.734738 18311 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:17:35.734762 18311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:35.742596 18311 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34527
I20260924 04:17:35.742620 18438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34527 every 8 connection(s)
I20260924 04:17:35.743460 18311 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:17:35.744601 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 18311
I20260924 04:17:35.744700 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:17:35.744939 392 external_mini_cluster.cc:1428] Setting key fe823ef7452a168d78b6dd8024c47d2b
I20260924 04:17:35.752138 18028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:35.745132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44801 (local address 127.0.98.62:43427)
0924 04:17:35.745272 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:35.745276 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:35.746373 (+ 1097us) server_negotiation.cc:407] Connection header received
0924 04:17:35.747867 (+ 1494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:35.747870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:35.747923 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:35.747991 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:17:35.748807 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:35.749328 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:35.750390 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:35.750606 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:35.750695 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:35.751691 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:17:35.751694 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:17:35.751697 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:17:35.751709 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:17:35.751728 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:17:35.751735 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:17:35.751804 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:35.751939 (+ 135us) server_negotiation.cc:299] Negotiation successful
0924 04:17:35.751999 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 04:17:35.752907 18441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:35.745559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43427 (local address 127.0.98.3:44801)
0924 04:17:35.746250 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:35.746293 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 04:17:35.747690 (+ 1397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:17:35.748146 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:17:35.748160 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:17:35.748240 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:17:35.748687 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:35.748698 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:35.749433 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:35.749437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:35.750284 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:35.750292 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:35.750730 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:35.750733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:35.750886 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:35.751446 (+ 560us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:17:35.751447 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:17:35.751488 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:17:35.751546 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:17:35.751814 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:17:35.751822 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:17:35.751828 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 04:17:35.751930 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 04:17:35.752204 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":112,"threads_started":1}
I20260924 04:17:35.753916 18439 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43427
I20260924 04:17:35.754132 18439 heartbeater.cc:464] Registering TS with master...
I20260924 04:17:35.754603 18439 heartbeater.cc:511] Master 127.0.98.62:43427 requested a full tablet report, sending...
I20260924 04:17:35.755560 17989 ts_manager.cc:194] Registered new tserver with Master: 8cddcc2c27af45eb8bd7db873a1059f7 (127.0.98.3:34527)
I20260924 04:17:35.756091 17989 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:44801
I20260924 04:17:35.758460 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:17:35.763410 18028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:35.760233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37634 (local address 127.0.98.62:43427)
0924 04:17:35.760349 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:35.760353 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:35.760508 (+ 155us) server_negotiation.cc:407] Connection header received
0924 04:17:35.760589 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:35.760592 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:35.760629 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:35.760692 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:17:35.760884 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:35.761380 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:35.762116 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:35.762325 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:35.762397 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:35.763063 (+ 666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:17:35.763066 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:17:35.763068 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:17:35.763077 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:17:35.763094 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:17:35.763101 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:17:35.763152 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:35.763213 (+ 61us) server_negotiation.cc:299] Negotiation successful
0924 04:17:35.763281 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 04:17:35.765933 17989 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37634:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:17:35.769032 17989 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:17:35.782421 18450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:35.776712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43865 (local address 127.0.0.1:34832)
0924 04:17:35.777068 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:35.777120 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 04:17:35.777275 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:17:35.778334 (+ 1059us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:17:35.778340 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:17:35.778365 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:17:35.778519 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:35.778526 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:35.779736 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:35.779739 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:35.780905 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:35.780914 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:35.781906 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:35.781909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:35.782011 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:35.782052 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 04:17:35.782147 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 04:17:35.782267 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":59,"threads_started":1}
I20260924 04:17:35.782673 18451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:35.776893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34832 (local address 127.0.98.2:43865)
0924 04:17:35.777615 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:35.777630 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:35.777656 (+ 26us) server_negotiation.cc:407] Connection header received
0924 04:17:35.777740 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:35.777746 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:35.777962 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:35.778140 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:17:35.778639 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:35.779637 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:35.781027 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:35.781798 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:35.781903 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:35.782360 (+ 457us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:35.782416 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 04:17:35.782514 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":101,"threads_started":1}
I20260924 04:17:35.782918 18453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:35.777205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34527 (local address 127.0.0.1:51124)
0924 04:17:35.778049 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:35.778069 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 04:17:35.778152 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:17:35.778703 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:17:35.778706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:17:35.778724 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:17:35.778813 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:35.778819 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:35.780250 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:35.780254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:35.781409 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:35.781420 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:35.782576 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:35.782579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:35.782665 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:35.782687 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 04:17:35.782752 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 04:17:35.782813 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":780,"thread_start_us":112,"threads_started":1}
I20260924 04:17:35.783407 18455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:35.777790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51124 (local address 127.0.98.3:34527)
0924 04:17:35.778266 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:35.778279 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:35.778303 (+ 24us) server_negotiation.cc:407] Connection header received
0924 04:17:35.778373 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:35.778378 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:35.778561 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:35.778706 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:17:35.778931 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:35.780103 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:35.781535 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:35.782257 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:35.782348 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:35.782514 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:35.783124 (+ 610us) server_negotiation.cc:299] Negotiation successful
0924 04:17:35.783257 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":138,"threads_started":1}
I20260924 04:17:35.783414 18452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:35.777348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37273 (local address 127.0.0.1:49960)
0924 04:17:35.778759 (+ 1411us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:35.778773 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:17:35.778863 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:17:35.779343 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:17:35.779346 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:17:35.779364 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:17:35.779441 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:35.779446 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:35.780808 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:35.780811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:35.781639 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:35.781648 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:35.783008 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:35.783012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:35.783114 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:35.783137 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 04:17:35.783204 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 04:17:35.783289 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1349,"mutex_wait_us":1,"thread_start_us":75,"threads_started":1}
I20260924 04:17:35.784461 18454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:35.777644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49960 (local address 127.0.98.1:37273)
0924 04:17:35.778081 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:35.778094 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:35.778857 (+ 763us) server_negotiation.cc:407] Connection header received
0924 04:17:35.779012 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:35.779017 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:35.779190 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:35.779366 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:17:35.779561 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:35.780697 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:35.781779 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:35.782503 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:35.782784 (+ 281us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:35.782950 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:35.783566 (+ 616us) server_negotiation.cc:299] Negotiation successful
0924 04:17:35.783670 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":90,"threads_started":1}
I20260924 04:17:35.786126 18373 tablet_service.cc:1467] Processing CreateTablet for tablet bbd8e4f401744e62898e41894ce1bb9d (DEFAULT_TABLE table=test-table [id=85d899e3a4c143a2add1ac3d97754454]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:17:35.785997 18242 tablet_service.cc:1467] Processing CreateTablet for tablet bbd8e4f401744e62898e41894ce1bb9d (DEFAULT_TABLE table=test-table [id=85d899e3a4c143a2add1ac3d97754454]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:17:35.787010 18373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbd8e4f401744e62898e41894ce1bb9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:17:35.787034 18242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbd8e4f401744e62898e41894ce1bb9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:17:35.786939 18111 tablet_service.cc:1467] Processing CreateTablet for tablet bbd8e4f401744e62898e41894ce1bb9d (DEFAULT_TABLE table=test-table [id=85d899e3a4c143a2add1ac3d97754454]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:17:35.787868 18111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbd8e4f401744e62898e41894ce1bb9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:17:35.793534 18456 tablet_bootstrap.cc:492] T bbd8e4f401744e62898e41894ce1bb9d P 8cddcc2c27af45eb8bd7db873a1059f7: Bootstrap starting.
I20260924 04:17:35.795109 18458 tablet_bootstrap.cc:492] T bbd8e4f401744e62898e41894ce1bb9d P 5c6ec328147f44acb5ac4e4ce0aa656f: Bootstrap starting.
I20260924 04:17:35.795517 18457 tablet_bootstrap.cc:492] T bbd8e4f401744e62898e41894ce1bb9d P 781be121ea5e4b4f9b8b3340b2d82958: Bootstrap starting.
I20260924 04:17:35.795676 18456 tablet_bootstrap.cc:654] T bbd8e4f401744e62898e41894ce1bb9d P 8cddcc2c27af45eb8bd7db873a1059f7: Neither blocks nor log segments found. Creating new log.
I20260924 04:17:35.796341 18456 log.cc:824] T bbd8e4f401744e62898e41894ce1bb9d P 8cddcc2c27af45eb8bd7db873a1059f7: Log is configured to *not* fsync() on all Append() calls
I20260924 04:17:35.797173 18458 tablet_bootstrap.cc:654] T bbd8e4f401744e62898e41894ce1bb9d P 5c6ec328147f44acb5ac4e4ce0aa656f: Neither blocks nor log segments found. Creating new log.
I20260924 04:17:35.797369 18457 tablet_bootstrap.cc:654] T bbd8e4f401744e62898e41894ce1bb9d P 781be121ea5e4b4f9b8b3340b2d82958: Neither blocks nor log segments found. Creating new log.
I20260924 04:17:35.798020 18457 log.cc:824] T bbd8e4f401744e62898e41894ce1bb9d P 781be121ea5e4b4f9b8b3340b2d82958: Log is configured to *not* fsync() on all Append() calls
I20260924 04:17:35.798020 18458 log.cc:824] T bbd8e4f401744e62898e41894ce1bb9d P 5c6ec328147f44acb5ac4e4ce0aa656f: Log is configured to *not* fsync() on all Append() calls
I20260924 04:17:35.799296 18456 tablet_bootstrap.cc:492] T bbd8e4f401744e62898e41894ce1bb9d P 8cddcc2c27af45eb8bd7db873a1059f7: No bootstrap required, opened a new log
I20260924 04:17:35.799474 18456 ts_tablet_manager.cc:1397] T bbd8e4f401744e62898e41894ce1bb9d P 8cddcc2c27af45eb8bd7db873a1059f7: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:17:35.799906 18458 tablet_bootstrap.cc:492] T bbd8e4f401744e62898e41894ce1bb9d P 5c6ec328147f44acb5ac4e4ce0aa656f: No bootstrap required, opened a new log
I20260924 04:17:35.799906 18457 tablet_bootstrap.cc:492] T bbd8e4f401744e62898e41894ce1bb9d P 781be121ea5e4b4f9b8b3340b2d82958: No bootstrap required, opened a new log
I20260924 04:17:35.800081 18458 ts_tablet_manager.cc:1397] T bbd8e4f401744e62898e41894ce1bb9d P 5c6ec328147f44acb5ac4e4ce0aa656f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 04:17:35.800081 18457 ts_tablet_manager.cc:1397] T bbd8e4f401744e62898e41894ce1bb9d P 781be121ea5e4b4f9b8b3340b2d82958: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 04:17:35.802573 18456 raft_consensus.cc:348] T bbd8e4f401744e62898e41894ce1bb9d P 8cddcc2c27af45eb8bd7db873a1059f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "781be121ea5e4b4f9b8b3340b2d82958" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43865 } } peers { permanent_uuid: "5c6ec328147f44acb5ac4e4ce0aa656f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37273 } } peers { permanent_uuid: "8cddcc2c27af45eb8bd7db873a1059f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34527 } }
I20260924 04:17:35.802784 18456 raft_consensus.cc:374] T bbd8e4f401744e62898e41894ce1bb9d P 8cddcc2c27af45eb8bd7db873a1059f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:17:35.802834 18456 raft_consensus.cc:729] T bbd8e4f401744e62898e41894ce1bb9d P 8cddcc2c27af45eb8bd7db873a1059f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cddcc2c27af45eb8bd7db873a1059f7, State: Initialized, Role: FOLLOWER
I20260924 04:17:35.803253 18456 consensus_queue.cc:260] T bbd8e4f401744e62898e41894ce1bb9d P 8cddcc2c27af45eb8bd7db873a1059f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "781be121ea5e4b4f9b8b3340b2d82958" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43865 } } peers { permanent_uuid: "5c6ec328147f44acb5ac4e4ce0aa656f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37273 } } peers { permanent_uuid: "8cddcc2c27af45eb8bd7db873a1059f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34527 } }
I20260924 04:17:35.803864 18439 heartbeater.cc:503] Master 127.0.98.62:43427 was elected leader, sending a full tablet report...
I20260924 04:17:35.804216 18456 ts_tablet_manager.cc:1428] T bbd8e4f401744e62898e41894ce1bb9d P 8cddcc2c27af45eb8bd7db873a1059f7: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260924 04:17:35.804142 18458 raft_consensus.cc:348] T bbd8e4f401744e62898e41894ce1bb9d P 5c6ec328147f44acb5ac4e4ce0aa656f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "781be121ea5e4b4f9b8b3340b2d82958" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43865 } } peers { permanent_uuid: "5c6ec328147f44acb5ac4e4ce0aa656f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37273 } } peers { permanent_uuid: "8cddcc2c27af45eb8bd7db873a1059f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34527 } }
I20260924 04:17:35.804142 18457 raft_consensus.cc:348] T bbd8e4f401744e62898e41894ce1bb9d P 781be121ea5e4b4f9b8b3340b2d82958 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "781be121ea5e4b4f9b8b3340b2d82958" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43865 } } peers { permanent_uuid: "5c6ec328147f44acb5ac4e4ce0aa656f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37273 } } peers { permanent_uuid: "8cddcc2c27af45eb8bd7db873a1059f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34527 } }
I20260924 04:17:35.804378 18457 raft_consensus.cc:374] T bbd8e4f401744e62898e41894ce1bb9d P 781be121ea5e4b4f9b8b3340b2d82958 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:17:35.804404 18458 raft_consensus.cc:374] T bbd8e4f401744e62898e41894ce1bb9d P 5c6ec328147f44acb5ac4e4ce0aa656f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:17:35.804447 18458 raft_consensus.cc:729] T bbd8e4f401744e62898e41894ce1bb9d P 5c6ec328147f44acb5ac4e4ce0aa656f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c6ec328147f44acb5ac4e4ce0aa656f, State: Initialized, Role: FOLLOWER
I20260924 04:17:35.804503 18457 raft_consensus.cc:729] T bbd8e4f401744e62898e41894ce1bb9d P 781be121ea5e4b4f9b8b3340b2d82958 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 781be121ea5e4b4f9b8b3340b2d82958, State: Initialized, Role: FOLLOWER
I20260924 04:17:35.805040 18457 consensus_queue.cc:260] T bbd8e4f401744e62898e41894ce1bb9d P 781be121ea5e4b4f9b8b3340b2d82958 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "781be121ea5e4b4f9b8b3340b2d82958" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43865 } } peers { permanent_uuid: "5c6ec328147f44acb5ac4e4ce0aa656f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37273 } } peers { permanent_uuid: "8cddcc2c27af45eb8bd7db873a1059f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34527 } }
I20260924 04:17:35.805685 18308 heartbeater.cc:503] Master 127.0.98.62:43427 was elected leader, sending a full tablet report...
I20260924 04:17:35.806002 18457 ts_tablet_manager.cc:1428] T bbd8e4f401744e62898e41894ce1bb9d P 781be121ea5e4b4f9b8b3340b2d82958: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 04:17:35.804926 18458 consensus_queue.cc:260] T bbd8e4f401744e62898e41894ce1bb9d P 5c6ec328147f44acb5ac4e4ce0aa656f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "781be121ea5e4b4f9b8b3340b2d82958" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43865 } } peers { permanent_uuid: "5c6ec328147f44acb5ac4e4ce0aa656f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37273 } } peers { permanent_uuid: "8cddcc2c27af45eb8bd7db873a1059f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34527 } }
I20260924 04:17:35.807303 18177 heartbeater.cc:503] Master 127.0.98.62:43427 was elected leader, sending a full tablet report...
I20260924 04:17:35.807559 18458 ts_tablet_manager.cc:1428] T bbd8e4f401744e62898e41894ce1bb9d P 5c6ec328147f44acb5ac4e4ce0aa656f: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
W20260924 04:17:35.821182 18309 tablet.cc:2369] T bbd8e4f401744e62898e41894ce1bb9d P 781be121ea5e4b4f9b8b3340b2d82958: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:17:35.889052 18178 tablet.cc:2369] T bbd8e4f401744e62898e41894ce1bb9d P 5c6ec328147f44acb5ac4e4ce0aa656f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:17:35.909374 18463 raft_consensus.cc:482] T bbd8e4f401744e62898e41894ce1bb9d P 781be121ea5e4b4f9b8b3340b2d82958 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:17:35.909592 18463 raft_consensus.cc:504] T bbd8e4f401744e62898e41894ce1bb9d P 781be121ea5e4b4f9b8b3340b2d82958 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "781be121ea5e4b4f9b8b3340b2d82958" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43865 } } peers { permanent_uuid: "5c6ec328147f44acb5ac4e4ce0aa656f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37273 } } peers { permanent_uuid: "8cddcc2c27af45eb8bd7db873a1059f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34527 } }
I20260924 04:17:35.910775 18463 leader_election.cc:288] T bbd8e4f401744e62898e41894ce1bb9d P 781be121ea5e4b4f9b8b3340b2d82958 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c6ec328147f44acb5ac4e4ce0aa656f (127.0.98.1:37273), 8cddcc2c27af45eb8bd7db873a1059f7 (127.0.98.3:34527)
I20260924 04:17:35.915469 18454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:35.911078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33523 (local address 127.0.98.1:37273)
0924 04:17:35.911205 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:35.911210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:35.911310 (+ 100us) server_negotiation.cc:407] Connection header received
0924 04:17:35.911451 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:35.911453 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:35.911500 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:35.911582 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:17:35.911856 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:35.912624 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:35.914287 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:35.914880 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:35.914981 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:35.915041 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:35.915283 (+ 242us) server_negotiation.cc:299] Negotiation successful
0924 04:17:35.915346 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 04:17:35.915469 18455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:35.911117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43443 (local address 127.0.98.3:34527)
0924 04:17:35.911260 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:35.911264 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:35.911525 (+ 261us) server_negotiation.cc:407] Connection header received
0924 04:17:35.911607 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:35.911610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:35.911653 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:35.911718 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:17:35.911969 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:35.912689 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:35.914287 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:35.914880 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:35.914981 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:35.915041 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:35.915269 (+ 228us) server_negotiation.cc:299] Negotiation successful
0924 04:17:35.915341 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 04:17:35.915486 18310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:35.910987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37273 (local address 127.0.98.2:33523)
0924 04:17:35.911204 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:35.911218 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:17:35.911346 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:17:35.911600 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:17:35.911611 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:17:35.911627 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:17:35.911734 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:35.911741 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:35.912746 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:35.912749 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:35.914172 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:35.914183 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:35.914997 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:35.915001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:35.915149 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:35.915174 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 04:17:35.915259 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 04:17:35.915339 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":205}
I20260924 04:17:35.915493 18465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:35.911024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34527 (local address 127.0.98.2:43443)
0924 04:17:35.911427 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:35.911440 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:17:35.911530 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:17:35.911746 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:17:35.911748 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:17:35.911773 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:17:35.911876 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:35.911881 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:35.912782 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:35.912785 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:35.914171 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:35.914183 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:35.914997 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:35.915001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:35.915121 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:35.915148 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 04:17:35.915249 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 04:17:35.915338 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":17,"thread_start_us":146,"threads_started":1}
I20260924 04:17:35.916186 18131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbd8e4f401744e62898e41894ce1bb9d" candidate_uuid: "781be121ea5e4b4f9b8b3340b2d82958" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c6ec328147f44acb5ac4e4ce0aa656f" is_pre_election: true
I20260924 04:17:35.916325 18393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbd8e4f401744e62898e41894ce1bb9d" candidate_uuid: "781be121ea5e4b4f9b8b3340b2d82958" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cddcc2c27af45eb8bd7db873a1059f7" is_pre_election: true
I20260924 04:17:35.916472 18131 raft_consensus.cc:2454] T bbd8e4f401744e62898e41894ce1bb9d P 5c6ec328147f44acb5ac4e4ce0aa656f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 781be121ea5e4b4f9b8b3340b2d82958 in term 0.
I20260924 04:17:35.916635 18393 raft_consensus.cc:2454] T bbd8e4f401744e62898e41894ce1bb9d P 8cddcc2c27af45eb8bd7db873a1059f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 781be121ea5e4b4f9b8b3340b2d82958 in term 0.
I20260924 04:17:35.916975 18198 leader_election.cc:302] T bbd8e4f401744e62898e41894ce1bb9d P 781be121ea5e4b4f9b8b3340b2d82958 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c6ec328147f44acb5ac4e4ce0aa656f, 781be121ea5e4b4f9b8b3340b2d82958; no voters:
I20260924 04:17:35.917285 18463 raft_consensus.cc:2781] T bbd8e4f401744e62898e41894ce1bb9d P 781be121ea5e4b4f9b8b3340b2d82958 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:17:35.917376 18463 raft_consensus.cc:482] T bbd8e4f401744e62898e41894ce1bb9d P 781be121ea5e4b4f9b8b3340b2d82958 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:17:35.917431 18463 raft_consensus.cc:3037] T bbd8e4f401744e62898e41894ce1bb9d P 781be121ea5e4b4f9b8b3340b2d82958 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:17:35.918529 18463 raft_consensus.cc:504] T bbd8e4f401744e62898e41894ce1bb9d P 781be121ea5e4b4f9b8b3340b2d82958 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "781be121ea5e4b4f9b8b3340b2d82958" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43865 } } peers { permanent_uuid: "5c6ec328147f44acb5ac4e4ce0aa656f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37273 } } peers { permanent_uuid: "8cddcc2c27af45eb8bd7db873a1059f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34527 } }
I20260924 04:17:35.918890 18463 leader_election.cc:288] T bbd8e4f401744e62898e41894ce1bb9d P 781be121ea5e4b4f9b8b3340b2d82958 [CANDIDATE]: Term 1 election: Requested vote from peers 5c6ec328147f44acb5ac4e4ce0aa656f (127.0.98.1:37273), 8cddcc2c27af45eb8bd7db873a1059f7 (127.0.98.3:34527)
I20260924 04:17:35.919250 18131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbd8e4f401744e62898e41894ce1bb9d" candidate_uuid: "781be121ea5e4b4f9b8b3340b2d82958" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c6ec328147f44acb5ac4e4ce0aa656f"
I20260924 04:17:35.919250 18393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbd8e4f401744e62898e41894ce1bb9d" candidate_uuid: "781be121ea5e4b4f9b8b3340b2d82958" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cddcc2c27af45eb8bd7db873a1059f7"
I20260924 04:17:35.919373 18131 raft_consensus.cc:3037] T bbd8e4f401744e62898e41894ce1bb9d P 5c6ec328147f44acb5ac4e4ce0aa656f [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:17:35.919377 18393 raft_consensus.cc:3037] T bbd8e4f401744e62898e41894ce1bb9d P 8cddcc2c27af45eb8bd7db873a1059f7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:17:35.920342 18131 raft_consensus.cc:2454] T bbd8e4f401744e62898e41894ce1bb9d P 5c6ec328147f44acb5ac4e4ce0aa656f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 781be121ea5e4b4f9b8b3340b2d82958 in term 1.
I20260924 04:17:35.920439 18393 raft_consensus.cc:2454] T bbd8e4f401744e62898e41894ce1bb9d P 8cddcc2c27af45eb8bd7db873a1059f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 781be121ea5e4b4f9b8b3340b2d82958 in term 1.
I20260924 04:17:35.920611 18198 leader_election.cc:302] T bbd8e4f401744e62898e41894ce1bb9d P 781be121ea5e4b4f9b8b3340b2d82958 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c6ec328147f44acb5ac4e4ce0aa656f, 781be121ea5e4b4f9b8b3340b2d82958; no voters:
I20260924 04:17:35.920785 18463 raft_consensus.cc:2781] T bbd8e4f401744e62898e41894ce1bb9d P 781be121ea5e4b4f9b8b3340b2d82958 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:17:35.921013 18463 raft_consensus.cc:686] T bbd8e4f401744e62898e41894ce1bb9d P 781be121ea5e4b4f9b8b3340b2d82958 [term 1 LEADER]: Becoming Leader. State: Replica: 781be121ea5e4b4f9b8b3340b2d82958, State: Running, Role: LEADER
I20260924 04:17:35.921325 18463 consensus_queue.cc:237] T bbd8e4f401744e62898e41894ce1bb9d P 781be121ea5e4b4f9b8b3340b2d82958 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "781be121ea5e4b4f9b8b3340b2d82958" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43865 } } peers { permanent_uuid: "5c6ec328147f44acb5ac4e4ce0aa656f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37273 } } peers { permanent_uuid: "8cddcc2c27af45eb8bd7db873a1059f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34527 } }
I20260924 04:17:35.924540 17988 catalog_manager.cc:5495] T bbd8e4f401744e62898e41894ce1bb9d P 781be121ea5e4b4f9b8b3340b2d82958 reported cstate change: term changed from 0 to 1, leader changed from <none> to 781be121ea5e4b4f9b8b3340b2d82958 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "781be121ea5e4b4f9b8b3340b2d82958" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "781be121ea5e4b4f9b8b3340b2d82958" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c6ec328147f44acb5ac4e4ce0aa656f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8cddcc2c27af45eb8bd7db873a1059f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34527 } health_report { overall_health: UNKNOWN } } }
I20260924 04:17:35.969570 18451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:35.966299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34848 (local address 127.0.98.2:43865)
0924 04:17:35.966409 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:35.966413 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:35.966427 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:17:35.966507 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:35.966509 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:35.966569 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:35.966670 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:17:35.966859 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:35.967507 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:35.968305 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:35.968524 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:35.968611 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:35.968824 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:17:35.969248 (+ 424us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:17:35.969336 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:35.969364 (+ 28us) server_negotiation.cc:299] Negotiation successful
0924 04:17:35.969409 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 04:17:35.975356 18131 raft_consensus.cc:1261] T bbd8e4f401744e62898e41894ce1bb9d P 5c6ec328147f44acb5ac4e4ce0aa656f [term 1 FOLLOWER]: Refusing update from remote peer 781be121ea5e4b4f9b8b3340b2d82958: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:17:35.975368 18393 raft_consensus.cc:1261] T bbd8e4f401744e62898e41894ce1bb9d P 8cddcc2c27af45eb8bd7db873a1059f7 [term 1 FOLLOWER]: Refusing update from remote peer 781be121ea5e4b4f9b8b3340b2d82958: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:17:35.976007 18463 consensus_queue.cc:1035] T bbd8e4f401744e62898e41894ce1bb9d P 781be121ea5e4b4f9b8b3340b2d82958 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c6ec328147f44acb5ac4e4ce0aa656f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:17:35.976212 18466 consensus_queue.cc:1035] T bbd8e4f401744e62898e41894ce1bb9d P 781be121ea5e4b4f9b8b3340b2d82958 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cddcc2c27af45eb8bd7db873a1059f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:17:35.986688 18471 mvcc.cc:205] Tried to move back new op lower bound from 7332755275667828736 to 7332755275456282624. Current Snapshot: MvccSnapshot[applied={T|T < 7332755275667828736}]
I20260924 04:17:35.991685 17988 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37634:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:17:35.991853 17988 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37634:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:17:35.994131 17988 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2f504f029beb41c38e474516a151de95: Sending DeleteTablet for 3 replicas of tablet bbd8e4f401744e62898e41894ce1bb9d
I20260924 04:17:35.994967 18111 tablet_service.cc:1514] Processing DeleteTablet for tablet bbd8e4f401744e62898e41894ce1bb9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:17:35 UTC) from {username='slave'} at 127.0.0.1:49960
I20260924 04:17:35.994961 18373 tablet_service.cc:1514] Processing DeleteTablet for tablet bbd8e4f401744e62898e41894ce1bb9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:17:35 UTC) from {username='slave'} at 127.0.0.1:51124
I20260924 04:17:35.995525 18479 tablet_replica.cc:330] T bbd8e4f401744e62898e41894ce1bb9d P 5c6ec328147f44acb5ac4e4ce0aa656f: stopping tablet replica
I20260924 04:17:35.995656 18480 tablet_replica.cc:330] T bbd8e4f401744e62898e41894ce1bb9d P 8cddcc2c27af45eb8bd7db873a1059f7: stopping tablet replica
I20260924 04:17:35.995611 18242 tablet_service.cc:1514] Processing DeleteTablet for tablet bbd8e4f401744e62898e41894ce1bb9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:17:35 UTC) from {username='slave'} at 127.0.0.1:34832
I20260924 04:17:35.995944 18479 raft_consensus.cc:2229] T bbd8e4f401744e62898e41894ce1bb9d P 5c6ec328147f44acb5ac4e4ce0aa656f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:17:35.995946 18480 raft_consensus.cc:2229] T bbd8e4f401744e62898e41894ce1bb9d P 8cddcc2c27af45eb8bd7db873a1059f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:17:35.996272 18479 raft_consensus.cc:2258] T bbd8e4f401744e62898e41894ce1bb9d P 5c6ec328147f44acb5ac4e4ce0aa656f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:17:35.996272 18480 raft_consensus.cc:2258] T bbd8e4f401744e62898e41894ce1bb9d P 8cddcc2c27af45eb8bd7db873a1059f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:17:35.996732 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 18049
I20260924 04:17:35.997416 18481 tablet_replica.cc:330] T bbd8e4f401744e62898e41894ce1bb9d P 781be121ea5e4b4f9b8b3340b2d82958: stopping tablet replica
I20260924 04:17:35.997705 18481 raft_consensus.cc:2229] T bbd8e4f401744e62898e41894ce1bb9d P 781be121ea5e4b4f9b8b3340b2d82958 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:17:35.998067 18481 raft_consensus.cc:2258] T bbd8e4f401744e62898e41894ce1bb9d P 781be121ea5e4b4f9b8b3340b2d82958 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:17:35.999287 18480 ts_tablet_manager.cc:1905] T bbd8e4f401744e62898e41894ce1bb9d P 8cddcc2c27af45eb8bd7db873a1059f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:17:35.999420 18481 ts_tablet_manager.cc:1905] T bbd8e4f401744e62898e41894ce1bb9d P 781be121ea5e4b4f9b8b3340b2d82958: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:17:36.002080 18480 ts_tablet_manager.cc:1918] T bbd8e4f401744e62898e41894ce1bb9d P 8cddcc2c27af45eb8bd7db873a1059f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:17:36.002080 18481 ts_tablet_manager.cc:1918] T bbd8e4f401744e62898e41894ce1bb9d P 781be121ea5e4b4f9b8b3340b2d82958: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:17:36.002187 18481 log.cc:1196] T bbd8e4f401744e62898e41894ce1bb9d P 781be121ea5e4b4f9b8b3340b2d82958: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/ts-1/wal/wals/bbd8e4f401744e62898e41894ce1bb9d
I20260924 04:17:36.002219 18480 log.cc:1196] T bbd8e4f401744e62898e41894ce1bb9d P 8cddcc2c27af45eb8bd7db873a1059f7: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790223216265354-392-0/minicluster-data/ts-2/wal/wals/bbd8e4f401744e62898e41894ce1bb9d
I20260924 04:17:36.002540 18480 ts_tablet_manager.cc:1939] T bbd8e4f401744e62898e41894ce1bb9d P 8cddcc2c27af45eb8bd7db873a1059f7: Deleting consensus metadata
I20260924 04:17:36.002542 18481 ts_tablet_manager.cc:1939] T bbd8e4f401744e62898e41894ce1bb9d P 781be121ea5e4b4f9b8b3340b2d82958: Deleting consensus metadata
I20260924 04:17:36.003803 17973 catalog_manager.cc:4841] TS 8cddcc2c27af45eb8bd7db873a1059f7 (127.0.98.3:34527): tablet bbd8e4f401744e62898e41894ce1bb9d (table test-table [id=85d899e3a4c143a2add1ac3d97754454]) successfully deleted
I20260924 04:17:36.005196 17976 catalog_manager.cc:4841] TS 781be121ea5e4b4f9b8b3340b2d82958 (127.0.98.2:43865): tablet bbd8e4f401744e62898e41894ce1bb9d (table test-table [id=85d899e3a4c143a2add1ac3d97754454]) successfully deleted
W20260924 04:17:36.005442 17976 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37273 (error 108)
I20260924 04:17:36.006055 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 18180
I20260924 04:17:36.006408 18453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:36.006104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37273 (local address 127.0.0.1:49962)
0924 04:17:36.006274 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:36.006331 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260924 04:17:36.011020 17976 catalog_manager.cc:4584] TS 5c6ec328147f44acb5ac4e4ce0aa656f (127.0.98.1:37273): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bbd8e4f401744e62898e41894ce1bb9d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37273: connect: Connection refused (error 111)
I20260924 04:17:36.011529 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 18311
I20260924 04:17:36.015981 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 17958
2026-09-24T04:17:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1075 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:17:36 dist-test-slave-p401 krb5kdc[18485](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:17:36 dist-test-slave-p401 krb5kdc[18485](info): set up 2 sockets
Sep 24 04:17:36 dist-test-slave-p401 krb5kdc[18485](info): commencing operation
krb5kdc: starting...
W20260924 04:17:36.966868 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/krb5kdc/test-user.keytab.
Sep 24 04:17:37 dist-test-slave-p401 krb5kdc[18485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:17:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:17:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:17:37.077910 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45367
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45367
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:17:37.183398 18498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:17:37.183665 18498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:17:37.183722 18498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:17:37.187743 18498 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:17:37.187796 18498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:17:37.187824 18498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:17:37.187844 18498 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:17:37.187870 18498 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:17:37.192976 18498 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:17:37.193958 18498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:17:37.194950 18498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:17:37.203043 18506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:17:37.203033 18503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:17:37.203027 18504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:17:37.203428 18498 server_base.cc:1034] running on GCE node
I20260924 04:17:37.203982 18498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:17:37.205072 18498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:17:37.206239 18498 hybrid_clock.cc:648] HybridClock initialized: now 1790223457206221 us; error 36 us; skew 500 ppm
Sep 24 04:17:37 dist-test-slave-p401 krb5kdc[18485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223457, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:17:37.209012 18498 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:17:37.209842 18498 webserver.cc:460] Webserver started at http://127.0.98.62:38153/ using document root <none> and password file <none>
I20260924 04:17:37.210367 18498 fs_manager.cc:359] Metadata directory not provided
I20260924 04:17:37.210423 18498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:17:37.210579 18498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:17:37.212602 18498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/master-0/data/instance:
uuid: "1a67dac6771a4652bdabebde55b52ce9"
format_stamp: "Formatted at 2026-09-24 04:17:37 on dist-test-slave-p401"
server_key: "e08edc7d9e2f45c0d482d10bee30cf44"
server_key_iv: "43000dd1547e57dde68afdddde6f20f1"
server_key_version: "encryptionkey@0"
I20260924 04:17:37.213109 18498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/master-0/wal/instance:
uuid: "1a67dac6771a4652bdabebde55b52ce9"
format_stamp: "Formatted at 2026-09-24 04:17:37 on dist-test-slave-p401"
server_key: "e08edc7d9e2f45c0d482d10bee30cf44"
server_key_iv: "43000dd1547e57dde68afdddde6f20f1"
server_key_version: "encryptionkey@0"
I20260924 04:17:37.216612 18498 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:17:37.218974 18513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:37.220230 18498 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:17:37.220331 18498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/master-0/wal
uuid: "1a67dac6771a4652bdabebde55b52ce9"
format_stamp: "Formatted at 2026-09-24 04:17:37 on dist-test-slave-p401"
server_key: "e08edc7d9e2f45c0d482d10bee30cf44"
server_key_iv: "43000dd1547e57dde68afdddde6f20f1"
server_key_version: "encryptionkey@0"
I20260924 04:17:37.220436 18498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:17:37.240154 18498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:17:37.243327 18498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:17:37.243503 18498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:17:37.250015 18498 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45367
I20260924 04:17:37.250029 18565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45367 every 8 connection(s)
I20260924 04:17:37.250982 18498 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:17:37.253129 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 18498
I20260924 04:17:37.253249 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/master-0/wal/instance
I20260924 04:17:37.253525 392 external_mini_cluster.cc:1428] Setting key caa4f657b4056feafea8fb21c41ae56e
I20260924 04:17:37.254372 18566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:17:37.260955 18566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a67dac6771a4652bdabebde55b52ce9: Bootstrap starting.
Sep 24 04:17:37 dist-test-slave-p401 krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:17:37.263182 18566 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a67dac6771a4652bdabebde55b52ce9: Neither blocks nor log segments found. Creating new log.
I20260924 04:17:37.264072 18566 log.cc:824] T 00000000000000000000000000000000 P 1a67dac6771a4652bdabebde55b52ce9: Log is configured to *not* fsync() on all Append() calls
I20260924 04:17:37.266335 18566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a67dac6771a4652bdabebde55b52ce9: No bootstrap required, opened a new log
I20260924 04:17:37.268476 18569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:37.254977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59660 (local address 127.0.98.62:45367)
0924 04:17:37.255398 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:37.255416 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:37.255457 (+ 41us) server_negotiation.cc:407] Connection header received
0924 04:17:37.256264 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:37.256285 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:37.256688 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:37.257056 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:17:37.257950 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:37.259422 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:37.260299 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:37.260961 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:37.261115 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:37.263461 (+ 2346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:17:37.263468 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:17:37.263485 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:17:37.263518 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:17:37.265211 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:17:37.265629 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:17:37.265634 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:17:37.265640 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:17:37.265776 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:17:37.266039 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:17:37.266042 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:17:37.266043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:17:37.266569 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:17:37.266754 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:37.266950 (+ 196us) server_negotiation.cc:299] Negotiation successful
0924 04:17:37.267286 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":115,"threads_started":1}
I20260924 04:17:37.270217 18566 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a67dac6771a4652bdabebde55b52ce9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a67dac6771a4652bdabebde55b52ce9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45367 } }
I20260924 04:17:37.270430 18566 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a67dac6771a4652bdabebde55b52ce9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:17:37.270468 18566 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a67dac6771a4652bdabebde55b52ce9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a67dac6771a4652bdabebde55b52ce9, State: Initialized, Role: FOLLOWER
I20260924 04:17:37.270897 18566 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a67dac6771a4652bdabebde55b52ce9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a67dac6771a4652bdabebde55b52ce9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45367 } }
I20260924 04:17:37.271021 18566 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a67dac6771a4652bdabebde55b52ce9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:17:37.271068 18566 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a67dac6771a4652bdabebde55b52ce9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:17:37.271135 18566 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a67dac6771a4652bdabebde55b52ce9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:17:37.272059 18566 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a67dac6771a4652bdabebde55b52ce9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a67dac6771a4652bdabebde55b52ce9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45367 } }
I20260924 04:17:37.272384 18566 leader_election.cc:302] T 00000000000000000000000000000000 P 1a67dac6771a4652bdabebde55b52ce9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a67dac6771a4652bdabebde55b52ce9; no voters:
I20260924 04:17:37.272719 18566 leader_election.cc:288] T 00000000000000000000000000000000 P 1a67dac6771a4652bdabebde55b52ce9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:17:37.272897 18571 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a67dac6771a4652bdabebde55b52ce9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:17:37.273104 18571 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a67dac6771a4652bdabebde55b52ce9 [term 1 LEADER]: Becoming Leader. State: Replica: 1a67dac6771a4652bdabebde55b52ce9, State: Running, Role: LEADER
I20260924 04:17:37.273408 18571 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a67dac6771a4652bdabebde55b52ce9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a67dac6771a4652bdabebde55b52ce9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45367 } }
I20260924 04:17:37.273877 18566 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a67dac6771a4652bdabebde55b52ce9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:17:37.274876 18573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a67dac6771a4652bdabebde55b52ce9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a67dac6771a4652bdabebde55b52ce9. Latest consensus state: current_term: 1 leader_uuid: "1a67dac6771a4652bdabebde55b52ce9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a67dac6771a4652bdabebde55b52ce9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45367 } } }
I20260924 04:17:37.275009 18573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a67dac6771a4652bdabebde55b52ce9 [sys.catalog]: This master's current role is: LEADER
I20260924 04:17:37.275328 18572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a67dac6771a4652bdabebde55b52ce9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a67dac6771a4652bdabebde55b52ce9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a67dac6771a4652bdabebde55b52ce9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45367 } } }
I20260924 04:17:37.275409 18580 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:17:37.275456 18572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a67dac6771a4652bdabebde55b52ce9 [sys.catalog]: This master's current role is: LEADER
I20260924 04:17:37.278551 18580 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:17:37.283787 18580 catalog_manager.cc:1347] Generated new cluster ID: d32f573b96824713b7f0926b0072ec8a
I20260924 04:17:37.283859 18580 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:17:37.294234 18580 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:17:37.295428 18580 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:17:37.303435 18580 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a67dac6771a4652bdabebde55b52ce9: Generated new TSK 0
I20260924 04:17:37.304097 18580 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:17:37.337872 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45367
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:17:37.447650 18592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:17:37.447937 18592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:17:37.447997 18592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:17:37.452107 18592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:17:37.452173 18592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:17:37.452252 18592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:17:37.457567 18592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:17:37.458604 18592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:17:37.459622 18592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:17:37.469005 18598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:17:37.469005 18597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:17:37.469022 18600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:17:37.469183 18592 server_base.cc:1034] running on GCE node
I20260924 04:17:37.469696 18592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:17:37.470311 18592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:17:37.471457 18592 hybrid_clock.cc:648] HybridClock initialized: now 1790223457471438 us; error 31 us; skew 500 ppm
Sep 24 04:17:37 dist-test-slave-p401 krb5kdc[18485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223457, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:17:37.474035 18592 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:17:37.474959 18592 webserver.cc:460] Webserver started at http://127.0.98.1:45407/ using document root <none> and password file <none>
I20260924 04:17:37.475591 18592 fs_manager.cc:359] Metadata directory not provided
I20260924 04:17:37.475660 18592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:17:37.475880 18592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:17:37.477937 18592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/ts-0/data/instance:
uuid: "179672a4c96e4581bae6eafee4a44f8f"
format_stamp: "Formatted at 2026-09-24 04:17:37 on dist-test-slave-p401"
server_key: "680bab0216eaad31307c5a4aa1ee7fab"
server_key_iv: "b43cbdc5324a64f3795cc7386ec1c093"
server_key_version: "encryptionkey@0"
I20260924 04:17:37.478499 18592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/ts-0/wal/instance:
uuid: "179672a4c96e4581bae6eafee4a44f8f"
format_stamp: "Formatted at 2026-09-24 04:17:37 on dist-test-slave-p401"
server_key: "680bab0216eaad31307c5a4aa1ee7fab"
server_key_iv: "b43cbdc5324a64f3795cc7386ec1c093"
server_key_version: "encryptionkey@0"
I20260924 04:17:37.482527 18592 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260924 04:17:37.485146 18607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:37.486464 18592 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:17:37.486574 18592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/ts-0/wal
uuid: "179672a4c96e4581bae6eafee4a44f8f"
format_stamp: "Formatted at 2026-09-24 04:17:37 on dist-test-slave-p401"
server_key: "680bab0216eaad31307c5a4aa1ee7fab"
server_key_iv: "b43cbdc5324a64f3795cc7386ec1c093"
server_key_version: "encryptionkey@0"
I20260924 04:17:37.486680 18592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:17:37.505294 18592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:17:37.508543 18592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:17:37.508734 18592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:17:37.509323 18592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:17:37.510130 18592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:17:37.510174 18592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:37.510226 18592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:17:37.510254 18592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:37.518507 18592 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42171
I20260924 04:17:37.518533 18720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42171 every 8 connection(s)
I20260924 04:17:37.519433 18592 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:17:37.524055 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 18592
I20260924 04:17:37.524170 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:17:37.524502 392 external_mini_cluster.cc:1428] Setting key 422181283cc0871b1a5670608bc45581
Sep 24 04:17:37 dist-test-slave-p401 krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223457, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:17:37.532493 18569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:37.521221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44065 (local address 127.0.98.62:45367)
0924 04:17:37.521368 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:37.521372 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:37.522125 (+ 753us) server_negotiation.cc:407] Connection header received
0924 04:17:37.522975 (+ 850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:37.522979 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:37.523025 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:37.523096 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:17:37.524390 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:37.525078 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:37.526448 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:37.526765 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:37.526864 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:37.529359 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:17:37.529362 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:17:37.529364 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:17:37.529387 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:17:37.530611 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:17:37.531066 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:17:37.531069 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:17:37.531070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:17:37.531140 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:17:37.531461 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:17:37.531464 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:17:37.531465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:17:37.531632 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:17:37.531722 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:37.532209 (+ 487us) server_negotiation.cc:299] Negotiation successful
0924 04:17:37.532333 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:17:37.533280 18723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:37.521526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.1:44065)
0924 04:17:37.521970 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:37.522013 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 04:17:37.522782 (+ 769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:17:37.523293 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:17:37.523302 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:17:37.523693 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:17:37.524200 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:37.524218 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:37.525195 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:37.525199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:37.526326 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:37.526335 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:37.526888 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:37.526891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:37.527032 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:37.527649 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:17:37.527678 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:17:37.529211 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:17:37.530714 (+ 1503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:17:37.530720 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:17:37.530734 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:17:37.530977 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:17:37.531218 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:17:37.531221 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:17:37.531223 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:17:37.531379 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:17:37.531730 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:17:37.531736 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:17:37.531981 (+ 245us) client_negotiation.cc:769] Sending connection context
0924 04:17:37.532134 (+ 153us) client_negotiation.cc:240] Negotiation successful
0924 04:17:37.532458 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":118,"threads_started":1}
I20260924 04:17:37.534412 18721 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45367
I20260924 04:17:37.534634 18721 heartbeater.cc:464] Registering TS with master...
I20260924 04:17:37.535130 18721 heartbeater.cc:511] Master 127.0.98.62:45367 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:17:37.536876 18530 ts_manager.cc:194] Registered new tserver with Master: 179672a4c96e4581bae6eafee4a44f8f (127.0.98.1:42171)
I20260924 04:17:37.538087 18530 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44065
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:17:37.549875 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45367
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:17:37.654210 18726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:17:37.654443 18726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:17:37.654481 18726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:17:37.658399 18726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:17:37.658453 18726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:17:37.658509 18726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:17:37.663527 18726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:17:37.664477 18726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:17:37.665462 18726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:17:37.674367 18731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:17:37.674495 18726 server_base.cc:1034] running on GCE node
W20260924 04:17:37.674371 18732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:17:37.674585 18734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:17:37.675096 18726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:17:37.675763 18726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:17:37.676945 18726 hybrid_clock.cc:648] HybridClock initialized: now 1790223457676916 us; error 41 us; skew 500 ppm
Sep 24 04:17:37 dist-test-slave-p401 krb5kdc[18485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223457, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:17:37.679544 18726 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:17:37.680352 18726 webserver.cc:460] Webserver started at http://127.0.98.2:43377/ using document root <none> and password file <none>
I20260924 04:17:37.680887 18726 fs_manager.cc:359] Metadata directory not provided
I20260924 04:17:37.680945 18726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:17:37.681135 18726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:17:37.683125 18726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/ts-1/data/instance:
uuid: "1c02948bfdcc4179b0a12a013c8e5468"
format_stamp: "Formatted at 2026-09-24 04:17:37 on dist-test-slave-p401"
server_key: "89831a392913471e2106758d4870451d"
server_key_iv: "0ebd69d8a4f646373a4d24241e22827a"
server_key_version: "encryptionkey@0"
I20260924 04:17:37.683705 18726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1c02948bfdcc4179b0a12a013c8e5468"
format_stamp: "Formatted at 2026-09-24 04:17:37 on dist-test-slave-p401"
server_key: "89831a392913471e2106758d4870451d"
server_key_iv: "0ebd69d8a4f646373a4d24241e22827a"
server_key_version: "encryptionkey@0"
I20260924 04:17:37.687296 18726 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:17:37.689613 18741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:37.690802 18726 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:17:37.690933 18726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/ts-1/wal
uuid: "1c02948bfdcc4179b0a12a013c8e5468"
format_stamp: "Formatted at 2026-09-24 04:17:37 on dist-test-slave-p401"
server_key: "89831a392913471e2106758d4870451d"
server_key_iv: "0ebd69d8a4f646373a4d24241e22827a"
server_key_version: "encryptionkey@0"
I20260924 04:17:37.691030 18726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:17:37.710397 18726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:17:37.713498 18726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:17:37.713678 18726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:17:37.714294 18726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:17:37.715099 18726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:17:37.715145 18726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:37.715197 18726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:17:37.715277 18726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:37.723274 18726 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34823
I20260924 04:17:37.723315 18854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34823 every 8 connection(s)
I20260924 04:17:37.724145 18726 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:17:37.725555 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 18726
I20260924 04:17:37.725656 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:17:37.725965 392 external_mini_cluster.cc:1428] Setting key a3a9301303396d340b2c5fa7625a6f37
Sep 24 04:17:37 dist-test-slave-p401 krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223457, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 04:17:37.737581 18569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:37.725810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45951 (local address 127.0.98.62:45367)
0924 04:17:37.725956 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:37.725960 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:37.727213 (+ 1253us) server_negotiation.cc:407] Connection header received
0924 04:17:37.728295 (+ 1082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:37.728298 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:37.728339 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:37.728402 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:17:37.729681 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:37.730435 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:37.731669 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:37.732014 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:37.732125 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:37.734578 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:17:37.734582 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:17:37.734585 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:17:37.734614 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:17:37.735930 (+ 1316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:17:37.736343 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:17:37.736346 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:17:37.736347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:17:37.736418 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:17:37.736672 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:17:37.736674 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:17:37.736675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:17:37.736830 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:17:37.736897 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:37.737302 (+ 405us) server_negotiation.cc:299] Negotiation successful
0924 04:17:37.737422 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:17:37.738391 18857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:37.726228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.2:45951)
0924 04:17:37.727085 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:37.727134 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 04:17:37.728071 (+ 937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:17:37.728574 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:17:37.728583 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:17:37.729037 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:17:37.729544 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:37.729557 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:37.730542 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:37.730545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:37.731564 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:37.731573 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:37.732164 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:37.732168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:37.732319 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:37.732887 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:17:37.732901 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:17:37.734424 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:17:37.736019 (+ 1595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:17:37.736023 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:17:37.736040 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:17:37.736276 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:17:37.736486 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:17:37.736488 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:17:37.736489 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:17:37.736602 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:17:37.736897 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:17:37.736901 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:17:37.737115 (+ 214us) client_negotiation.cc:769] Sending connection context
0924 04:17:37.737264 (+ 149us) client_negotiation.cc:240] Negotiation successful
0924 04:17:37.737518 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":109,"threads_started":1}
I20260924 04:17:37.739513 18855 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45367
I20260924 04:17:37.739732 18855 heartbeater.cc:464] Registering TS with master...
I20260924 04:17:37.740182 18855 heartbeater.cc:511] Master 127.0.98.62:45367 requested a full tablet report, sending...
I20260924 04:17:37.741156 18530 ts_manager.cc:194] Registered new tserver with Master: 1c02948bfdcc4179b0a12a013c8e5468 (127.0.98.2:34823)
I20260924 04:17:37.741694 18530 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45951
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:17:37.751726 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45367
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:17:37.859130 18860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:17:37.859453 18860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:17:37.859514 18860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:17:37.863410 18860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:17:37.863469 18860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:17:37.863548 18860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:17:37.868702 18860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:17:37.869691 18860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:17:37.870688 18860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:17:37.879804 18865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:17:37.879891 18860 server_base.cc:1034] running on GCE node
W20260924 04:17:37.879804 18866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:17:37.879858 18868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:17:37.880414 18860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:17:37.880999 18860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:17:37.882153 18860 hybrid_clock.cc:648] HybridClock initialized: now 1790223457882134 us; error 32 us; skew 500 ppm
Sep 24 04:17:37 dist-test-slave-p401 krb5kdc[18485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223457, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:17:37.890111 18860 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:17:37.890934 18860 webserver.cc:460] Webserver started at http://127.0.98.3:44973/ using document root <none> and password file <none>
I20260924 04:17:37.891499 18860 fs_manager.cc:359] Metadata directory not provided
I20260924 04:17:37.891562 18860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:17:37.891727 18860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:17:37.893633 18860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/ts-2/data/instance:
uuid: "21a9624e47ec4741979c59b360a31630"
format_stamp: "Formatted at 2026-09-24 04:17:37 on dist-test-slave-p401"
server_key: "c3797b1f0b2bfbf46066c5c165b23669"
server_key_iv: "5563a8c505be8f4e2a2d0b14e119f396"
server_key_version: "encryptionkey@0"
I20260924 04:17:37.894102 18860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/ts-2/wal/instance:
uuid: "21a9624e47ec4741979c59b360a31630"
format_stamp: "Formatted at 2026-09-24 04:17:37 on dist-test-slave-p401"
server_key: "c3797b1f0b2bfbf46066c5c165b23669"
server_key_iv: "5563a8c505be8f4e2a2d0b14e119f396"
server_key_version: "encryptionkey@0"
I20260924 04:17:37.897519 18860 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:17:37.899727 18875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:37.900854 18860 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:17:37.900940 18860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/ts-2/wal
uuid: "21a9624e47ec4741979c59b360a31630"
format_stamp: "Formatted at 2026-09-24 04:17:37 on dist-test-slave-p401"
server_key: "c3797b1f0b2bfbf46066c5c165b23669"
server_key_iv: "5563a8c505be8f4e2a2d0b14e119f396"
server_key_version: "encryptionkey@0"
I20260924 04:17:37.901031 18860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:17:37.914590 18860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:17:37.917254 18860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:17:37.917411 18860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:17:37.917944 18860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:17:37.918706 18860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:17:37.918751 18860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:37.918792 18860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:17:37.918820 18860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:37.926918 18860 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44533
I20260924 04:17:37.926957 18988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44533 every 8 connection(s)
I20260924 04:17:37.927754 18860 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 04:17:37 dist-test-slave-p401 krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223457, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:17:37.937062 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 18860
I20260924 04:17:37.937191 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:17:37.937461 392 external_mini_cluster.cc:1428] Setting key e95351352101d1de4a4cefeb4f981c43
I20260924 04:17:37.939608 18569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:37.929382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58015 (local address 127.0.98.62:45367)
0924 04:17:37.929500 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:37.929503 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:37.930266 (+ 763us) server_negotiation.cc:407] Connection header received
0924 04:17:37.931064 (+ 798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:37.931067 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:37.931114 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:37.931203 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:17:37.932278 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:37.932779 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:37.933896 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:37.934096 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:37.934169 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:37.936340 (+ 2171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:17:37.936343 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:17:37.936345 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:17:37.936367 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:17:37.937591 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:17:37.938147 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:17:37.938150 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:17:37.938152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:17:37.938248 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:17:37.938589 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:17:37.938592 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:17:37.938594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:17:37.938786 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:17:37.938875 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:37.939342 (+ 467us) server_negotiation.cc:299] Negotiation successful
0924 04:17:37.939458 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 04:17:37.940372 18991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:37.929711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.3:58015)
0924 04:17:37.930115 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:37.930154 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 04:17:37.930883 (+ 729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:17:37.931366 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:17:37.931373 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:17:37.931715 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:17:37.932172 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:37.932184 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:37.932917 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:37.932920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:37.933789 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:37.933807 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:37.934199 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:37.934201 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:37.934335 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:37.934797 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:17:37.934812 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:17:37.936187 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:17:37.937687 (+ 1500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:17:37.937692 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:17:37.937722 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:17:37.937955 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:17:37.938318 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:17:37.938321 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:17:37.938322 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:17:37.938492 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:17:37.938884 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:17:37.938890 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:17:37.939106 (+ 216us) client_negotiation.cc:769] Sending connection context
0924 04:17:37.939299 (+ 193us) client_negotiation.cc:240] Negotiation successful
0924 04:17:37.939592 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":102,"threads_started":1}
I20260924 04:17:37.941320 18989 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45367
I20260924 04:17:37.941522 18989 heartbeater.cc:464] Registering TS with master...
I20260924 04:17:37.941951 18989 heartbeater.cc:511] Master 127.0.98.62:45367 requested a full tablet report, sending...
I20260924 04:17:37.942821 18530 ts_manager.cc:194] Registered new tserver with Master: 21a9624e47ec4741979c59b360a31630 (127.0.98.3:44533)
I20260924 04:17:37.943418 18530 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58015
I20260924 04:17:37.950719 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:17:37.958904 18569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:37.952459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59666 (local address 127.0.98.62:45367)
0924 04:17:37.952577 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:37.952582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:37.952729 (+ 147us) server_negotiation.cc:407] Connection header received
0924 04:17:37.952814 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:37.952816 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:37.952853 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:37.952913 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:17:37.953502 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:37.954024 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:37.954775 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:37.954978 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:37.955049 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:37.956443 (+ 1394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:17:37.956446 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:17:37.956448 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:17:37.956468 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:17:37.957538 (+ 1070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:17:37.957952 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:17:37.957954 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:17:37.957956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:17:37.958018 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:17:37.958232 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:17:37.958234 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:17:37.958235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:17:37.958416 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:17:37.958530 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:37.958684 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 04:17:37.958755 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 04:17:37.961689 18530 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59666:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:17:37.964658 18530 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:17:37.978765 19001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:37.972647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42171 (local address 127.0.0.1:45908)
0924 04:17:37.973409 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:37.973458 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 04:17:37.973580 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:17:37.974323 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:17:37.974326 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:17:37.974340 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:17:37.974409 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:37.974415 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:37.975929 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:37.975932 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:37.977227 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:37.977237 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:37.978216 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:37.978219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:37.978326 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:37.978361 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 04:17:37.978468 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 04:17:37.978585 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"mutex_wait_us":86,"thread_start_us":69,"threads_started":1}
I20260924 04:17:37.978777 19002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:37.972818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45908 (local address 127.0.98.1:42171)
0924 04:17:37.973141 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:37.973154 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:37.973729 (+ 575us) server_negotiation.cc:407] Connection header received
0924 04:17:37.973813 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:37.973818 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:37.974017 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:37.974158 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:17:37.974519 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:37.975811 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:37.977339 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:37.978089 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:37.978201 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:37.978356 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:37.978511 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 04:17:37.978608 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":135,"threads_started":1}
I20260924 04:17:37.978955 19000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:37.972723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34823 (local address 127.0.0.1:38750)
0924 04:17:37.973475 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:37.973489 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:17:37.973581 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:17:37.974218 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:17:37.974224 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:17:37.974245 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:17:37.974372 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:37.974378 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:37.975962 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:37.975965 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:37.977450 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:37.977460 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:37.978595 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:37.978599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:37.978683 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:37.978706 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 04:17:37.978772 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 04:17:37.978839 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"mutex_wait_us":141,"thread_start_us":93,"threads_started":1}
I20260924 04:17:37.979125 19003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:37.973328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44533 (local address 127.0.0.1:54114)
0924 04:17:37.974330 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:37.974342 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:17:37.974474 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:17:37.974870 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:17:37.974873 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:17:37.974888 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:17:37.974969 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:37.974974 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:37.976040 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:37.976043 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:37.977450 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:37.977461 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:37.978434 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:37.978438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:37.978515 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:37.978533 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 04:17:37.978589 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 04:17:37.978641 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":946,"mutex_wait_us":120,"thread_start_us":76,"threads_started":1}
I20260924 04:17:37.979429 19004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:37.973330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38750 (local address 127.0.98.2:34823)
0924 04:17:37.973772 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:37.973783 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:37.973800 (+ 17us) server_negotiation.cc:407] Connection header received
0924 04:17:37.973859 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:37.973863 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:37.974016 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:37.974490 (+ 474us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:17:37.974638 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:37.975797 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:37.977570 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:37.978326 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:37.978421 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:37.978538 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:37.979184 (+ 646us) server_negotiation.cc:299] Negotiation successful
0924 04:17:37.979316 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":133,"threads_started":1}
I20260924 04:17:37.982208 18789 tablet_service.cc:1467] Processing CreateTablet for tablet e101ffb35f7d4dff9f7a44ce6ed9650f (DEFAULT_TABLE table=test-table [id=7a6bd98d541247bd847921c564c3b465]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:17:37.982208 18655 tablet_service.cc:1467] Processing CreateTablet for tablet e101ffb35f7d4dff9f7a44ce6ed9650f (DEFAULT_TABLE table=test-table [id=7a6bd98d541247bd847921c564c3b465]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:17:37.983390 18789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e101ffb35f7d4dff9f7a44ce6ed9650f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:17:37.983390 18655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e101ffb35f7d4dff9f7a44ce6ed9650f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:17:37.983610 19005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:37.973706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54114 (local address 127.0.98.3:44533)
0924 04:17:37.974167 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:37.974201 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:37.974410 (+ 209us) server_negotiation.cc:407] Connection header received
0924 04:17:37.974599 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:37.974603 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:37.974748 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:37.974874 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:17:37.975093 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:37.975923 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:37.977590 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:37.978116 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:37.978243 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:37.978378 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:37.978718 (+ 340us) server_negotiation.cc:299] Negotiation successful
0924 04:17:37.978808 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":128,"threads_started":1}
I20260924 04:17:37.986068 18923 tablet_service.cc:1467] Processing CreateTablet for tablet e101ffb35f7d4dff9f7a44ce6ed9650f (DEFAULT_TABLE table=test-table [id=7a6bd98d541247bd847921c564c3b465]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:17:37.987036 18923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e101ffb35f7d4dff9f7a44ce6ed9650f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:17:37.990679 19006 tablet_bootstrap.cc:492] T e101ffb35f7d4dff9f7a44ce6ed9650f P 179672a4c96e4581bae6eafee4a44f8f: Bootstrap starting.
I20260924 04:17:37.991544 19007 tablet_bootstrap.cc:492] T e101ffb35f7d4dff9f7a44ce6ed9650f P 1c02948bfdcc4179b0a12a013c8e5468: Bootstrap starting.
I20260924 04:17:37.993208 19006 tablet_bootstrap.cc:654] T e101ffb35f7d4dff9f7a44ce6ed9650f P 179672a4c96e4581bae6eafee4a44f8f: Neither blocks nor log segments found. Creating new log.
I20260924 04:17:37.993289 19007 tablet_bootstrap.cc:654] T e101ffb35f7d4dff9f7a44ce6ed9650f P 1c02948bfdcc4179b0a12a013c8e5468: Neither blocks nor log segments found. Creating new log.
I20260924 04:17:37.993899 19008 tablet_bootstrap.cc:492] T e101ffb35f7d4dff9f7a44ce6ed9650f P 21a9624e47ec4741979c59b360a31630: Bootstrap starting.
I20260924 04:17:37.994501 19007 log.cc:824] T e101ffb35f7d4dff9f7a44ce6ed9650f P 1c02948bfdcc4179b0a12a013c8e5468: Log is configured to *not* fsync() on all Append() calls
I20260924 04:17:37.995381 19006 log.cc:824] T e101ffb35f7d4dff9f7a44ce6ed9650f P 179672a4c96e4581bae6eafee4a44f8f: Log is configured to *not* fsync() on all Append() calls
I20260924 04:17:37.996049 19008 tablet_bootstrap.cc:654] T e101ffb35f7d4dff9f7a44ce6ed9650f P 21a9624e47ec4741979c59b360a31630: Neither blocks nor log segments found. Creating new log.
I20260924 04:17:37.996639 19007 tablet_bootstrap.cc:492] T e101ffb35f7d4dff9f7a44ce6ed9650f P 1c02948bfdcc4179b0a12a013c8e5468: No bootstrap required, opened a new log
I20260924 04:17:37.996831 19007 ts_tablet_manager.cc:1397] T e101ffb35f7d4dff9f7a44ce6ed9650f P 1c02948bfdcc4179b0a12a013c8e5468: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 04:17:37.997144 19006 tablet_bootstrap.cc:492] T e101ffb35f7d4dff9f7a44ce6ed9650f P 179672a4c96e4581bae6eafee4a44f8f: No bootstrap required, opened a new log
I20260924 04:17:37.997150 19008 log.cc:824] T e101ffb35f7d4dff9f7a44ce6ed9650f P 21a9624e47ec4741979c59b360a31630: Log is configured to *not* fsync() on all Append() calls
I20260924 04:17:37.997325 19006 ts_tablet_manager.cc:1397] T e101ffb35f7d4dff9f7a44ce6ed9650f P 179672a4c96e4581bae6eafee4a44f8f: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260924 04:17:37.999015 19008 tablet_bootstrap.cc:492] T e101ffb35f7d4dff9f7a44ce6ed9650f P 21a9624e47ec4741979c59b360a31630: No bootstrap required, opened a new log
I20260924 04:17:37.999195 19008 ts_tablet_manager.cc:1397] T e101ffb35f7d4dff9f7a44ce6ed9650f P 21a9624e47ec4741979c59b360a31630: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 04:17:38.000279 19006 raft_consensus.cc:348] T e101ffb35f7d4dff9f7a44ce6ed9650f P 179672a4c96e4581bae6eafee4a44f8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179672a4c96e4581bae6eafee4a44f8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42171 } } peers { permanent_uuid: "1c02948bfdcc4179b0a12a013c8e5468" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34823 } } peers { permanent_uuid: "21a9624e47ec4741979c59b360a31630" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44533 } }
I20260924 04:17:38.000551 19006 raft_consensus.cc:374] T e101ffb35f7d4dff9f7a44ce6ed9650f P 179672a4c96e4581bae6eafee4a44f8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:17:38.000610 19006 raft_consensus.cc:729] T e101ffb35f7d4dff9f7a44ce6ed9650f P 179672a4c96e4581bae6eafee4a44f8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 179672a4c96e4581bae6eafee4a44f8f, State: Initialized, Role: FOLLOWER
I20260924 04:17:38.000557 19007 raft_consensus.cc:348] T e101ffb35f7d4dff9f7a44ce6ed9650f P 1c02948bfdcc4179b0a12a013c8e5468 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179672a4c96e4581bae6eafee4a44f8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42171 } } peers { permanent_uuid: "1c02948bfdcc4179b0a12a013c8e5468" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34823 } } peers { permanent_uuid: "21a9624e47ec4741979c59b360a31630" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44533 } }
I20260924 04:17:38.000823 19007 raft_consensus.cc:374] T e101ffb35f7d4dff9f7a44ce6ed9650f P 1c02948bfdcc4179b0a12a013c8e5468 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:17:38.000877 19007 raft_consensus.cc:729] T e101ffb35f7d4dff9f7a44ce6ed9650f P 1c02948bfdcc4179b0a12a013c8e5468 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c02948bfdcc4179b0a12a013c8e5468, State: Initialized, Role: FOLLOWER
I20260924 04:17:38.001320 19007 consensus_queue.cc:260] T e101ffb35f7d4dff9f7a44ce6ed9650f P 1c02948bfdcc4179b0a12a013c8e5468 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179672a4c96e4581bae6eafee4a44f8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42171 } } peers { permanent_uuid: "1c02948bfdcc4179b0a12a013c8e5468" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34823 } } peers { permanent_uuid: "21a9624e47ec4741979c59b360a31630" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44533 } }
I20260924 04:17:38.001231 19006 consensus_queue.cc:260] T e101ffb35f7d4dff9f7a44ce6ed9650f P 179672a4c96e4581bae6eafee4a44f8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179672a4c96e4581bae6eafee4a44f8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42171 } } peers { permanent_uuid: "1c02948bfdcc4179b0a12a013c8e5468" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34823 } } peers { permanent_uuid: "21a9624e47ec4741979c59b360a31630" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44533 } }
I20260924 04:17:38.002211 18721 heartbeater.cc:503] Master 127.0.98.62:45367 was elected leader, sending a full tablet report...
I20260924 04:17:38.002447 19006 ts_tablet_manager.cc:1428] T e101ffb35f7d4dff9f7a44ce6ed9650f P 179672a4c96e4581bae6eafee4a44f8f: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260924 04:17:38.002730 19008 raft_consensus.cc:348] T e101ffb35f7d4dff9f7a44ce6ed9650f P 21a9624e47ec4741979c59b360a31630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179672a4c96e4581bae6eafee4a44f8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42171 } } peers { permanent_uuid: "1c02948bfdcc4179b0a12a013c8e5468" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34823 } } peers { permanent_uuid: "21a9624e47ec4741979c59b360a31630" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44533 } }
I20260924 04:17:38.002977 19008 raft_consensus.cc:374] T e101ffb35f7d4dff9f7a44ce6ed9650f P 21a9624e47ec4741979c59b360a31630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:17:38.003031 19008 raft_consensus.cc:729] T e101ffb35f7d4dff9f7a44ce6ed9650f P 21a9624e47ec4741979c59b360a31630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21a9624e47ec4741979c59b360a31630, State: Initialized, Role: FOLLOWER
I20260924 04:17:38.003427 19007 ts_tablet_manager.cc:1428] T e101ffb35f7d4dff9f7a44ce6ed9650f P 1c02948bfdcc4179b0a12a013c8e5468: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260924 04:17:38.003433 19008 consensus_queue.cc:260] T e101ffb35f7d4dff9f7a44ce6ed9650f P 21a9624e47ec4741979c59b360a31630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179672a4c96e4581bae6eafee4a44f8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42171 } } peers { permanent_uuid: "1c02948bfdcc4179b0a12a013c8e5468" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34823 } } peers { permanent_uuid: "21a9624e47ec4741979c59b360a31630" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44533 } }
I20260924 04:17:38.003755 18855 heartbeater.cc:503] Master 127.0.98.62:45367 was elected leader, sending a full tablet report...
I20260924 04:17:38.004235 19008 ts_tablet_manager.cc:1428] T e101ffb35f7d4dff9f7a44ce6ed9650f P 21a9624e47ec4741979c59b360a31630: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260924 04:17:38.004506 18989 heartbeater.cc:503] Master 127.0.98.62:45367 was elected leader, sending a full tablet report...
W20260924 04:17:38.020609 18722 tablet.cc:2369] T e101ffb35f7d4dff9f7a44ce6ed9650f P 179672a4c96e4581bae6eafee4a44f8f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:17:38.030186 19012 raft_consensus.cc:482] T e101ffb35f7d4dff9f7a44ce6ed9650f P 1c02948bfdcc4179b0a12a013c8e5468 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:17:38.030382 19012 raft_consensus.cc:504] T e101ffb35f7d4dff9f7a44ce6ed9650f P 1c02948bfdcc4179b0a12a013c8e5468 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179672a4c96e4581bae6eafee4a44f8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42171 } } peers { permanent_uuid: "1c02948bfdcc4179b0a12a013c8e5468" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34823 } } peers { permanent_uuid: "21a9624e47ec4741979c59b360a31630" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44533 } }
I20260924 04:17:38.031534 19012 leader_election.cc:288] T e101ffb35f7d4dff9f7a44ce6ed9650f P 1c02948bfdcc4179b0a12a013c8e5468 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 179672a4c96e4581bae6eafee4a44f8f (127.0.98.1:42171), 21a9624e47ec4741979c59b360a31630 (127.0.98.3:44533)
I20260924 04:17:38.035579 18857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:38.031780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42171 (local address 127.0.98.2:56345)
0924 04:17:38.031932 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:38.031944 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:17:38.032019 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:17:38.032314 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:17:38.032317 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:17:38.032338 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:17:38.032458 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:38.032465 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:38.033456 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:38.033461 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:38.034503 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:38.034512 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:38.035154 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:38.035158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:38.035289 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:38.035315 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 04:17:38.035382 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 04:17:38.035450 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260924 04:17:38.035597 19002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:38.031844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56345 (local address 127.0.98.1:42171)
0924 04:17:38.031982 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:38.031986 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:38.032002 (+ 16us) server_negotiation.cc:407] Connection header received
0924 04:17:38.032153 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:38.032156 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:38.032220 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:38.032290 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:17:38.032577 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:38.033339 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:38.034620 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:38.035045 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:38.035125 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:38.035182 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:38.035396 (+ 214us) server_negotiation.cc:299] Negotiation successful
0924 04:17:38.035472 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 04:17:38.035784 19015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:38.031823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44533 (local address 127.0.98.2:56999)
0924 04:17:38.032148 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:38.032159 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:17:38.032230 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:17:38.032453 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:17:38.032455 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:17:38.032470 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:17:38.032580 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:38.032586 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:38.033692 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:38.033695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:38.034631 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:38.034641 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:38.035394 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:38.035398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:38.035496 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:38.035518 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 04:17:38.035589 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 04:17:38.035651 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":135,"threads_started":1}
I20260924 04:17:38.035792 19005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:38.032105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56999 (local address 127.0.98.3:44533)
0924 04:17:38.032222 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:38.032225 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:38.032236 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:17:38.032313 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:38.032316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:38.032359 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:38.032439 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:17:38.032884 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:38.033566 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:38.034774 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:38.035290 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:38.035391 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:38.035445 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:38.035614 (+ 169us) server_negotiation.cc:299] Negotiation successful
0924 04:17:38.035685 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 04:17:38.036231 18675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e101ffb35f7d4dff9f7a44ce6ed9650f" candidate_uuid: "1c02948bfdcc4179b0a12a013c8e5468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "179672a4c96e4581bae6eafee4a44f8f" is_pre_election: true
I20260924 04:17:38.036381 18943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e101ffb35f7d4dff9f7a44ce6ed9650f" candidate_uuid: "1c02948bfdcc4179b0a12a013c8e5468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21a9624e47ec4741979c59b360a31630" is_pre_election: true
I20260924 04:17:38.036506 18675 raft_consensus.cc:2454] T e101ffb35f7d4dff9f7a44ce6ed9650f P 179672a4c96e4581bae6eafee4a44f8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c02948bfdcc4179b0a12a013c8e5468 in term 0.
I20260924 04:17:38.036650 18943 raft_consensus.cc:2454] T e101ffb35f7d4dff9f7a44ce6ed9650f P 21a9624e47ec4741979c59b360a31630 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c02948bfdcc4179b0a12a013c8e5468 in term 0.
I20260924 04:17:38.036959 18745 leader_election.cc:302] T e101ffb35f7d4dff9f7a44ce6ed9650f P 1c02948bfdcc4179b0a12a013c8e5468 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 179672a4c96e4581bae6eafee4a44f8f, 1c02948bfdcc4179b0a12a013c8e5468; no voters:
I20260924 04:17:38.037204 19012 raft_consensus.cc:2781] T e101ffb35f7d4dff9f7a44ce6ed9650f P 1c02948bfdcc4179b0a12a013c8e5468 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:17:38.037281 19012 raft_consensus.cc:482] T e101ffb35f7d4dff9f7a44ce6ed9650f P 1c02948bfdcc4179b0a12a013c8e5468 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:17:38.037326 19012 raft_consensus.cc:3037] T e101ffb35f7d4dff9f7a44ce6ed9650f P 1c02948bfdcc4179b0a12a013c8e5468 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:17:38.038260 19012 raft_consensus.cc:504] T e101ffb35f7d4dff9f7a44ce6ed9650f P 1c02948bfdcc4179b0a12a013c8e5468 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179672a4c96e4581bae6eafee4a44f8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42171 } } peers { permanent_uuid: "1c02948bfdcc4179b0a12a013c8e5468" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34823 } } peers { permanent_uuid: "21a9624e47ec4741979c59b360a31630" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44533 } }
I20260924 04:17:38.038556 19012 leader_election.cc:288] T e101ffb35f7d4dff9f7a44ce6ed9650f P 1c02948bfdcc4179b0a12a013c8e5468 [CANDIDATE]: Term 1 election: Requested vote from peers 179672a4c96e4581bae6eafee4a44f8f (127.0.98.1:42171), 21a9624e47ec4741979c59b360a31630 (127.0.98.3:44533)
I20260924 04:17:38.038866 18675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e101ffb35f7d4dff9f7a44ce6ed9650f" candidate_uuid: "1c02948bfdcc4179b0a12a013c8e5468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "179672a4c96e4581bae6eafee4a44f8f"
I20260924 04:17:38.038868 18943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e101ffb35f7d4dff9f7a44ce6ed9650f" candidate_uuid: "1c02948bfdcc4179b0a12a013c8e5468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21a9624e47ec4741979c59b360a31630"
I20260924 04:17:38.038987 18943 raft_consensus.cc:3037] T e101ffb35f7d4dff9f7a44ce6ed9650f P 21a9624e47ec4741979c59b360a31630 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:17:38.038987 18675 raft_consensus.cc:3037] T e101ffb35f7d4dff9f7a44ce6ed9650f P 179672a4c96e4581bae6eafee4a44f8f [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:17:38.040201 18675 raft_consensus.cc:2454] T e101ffb35f7d4dff9f7a44ce6ed9650f P 179672a4c96e4581bae6eafee4a44f8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c02948bfdcc4179b0a12a013c8e5468 in term 1.
I20260924 04:17:38.040206 18943 raft_consensus.cc:2454] T e101ffb35f7d4dff9f7a44ce6ed9650f P 21a9624e47ec4741979c59b360a31630 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c02948bfdcc4179b0a12a013c8e5468 in term 1.
I20260924 04:17:38.040477 18745 leader_election.cc:302] T e101ffb35f7d4dff9f7a44ce6ed9650f P 1c02948bfdcc4179b0a12a013c8e5468 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 179672a4c96e4581bae6eafee4a44f8f, 1c02948bfdcc4179b0a12a013c8e5468, 21a9624e47ec4741979c59b360a31630; no voters:
I20260924 04:17:38.040635 19012 raft_consensus.cc:2781] T e101ffb35f7d4dff9f7a44ce6ed9650f P 1c02948bfdcc4179b0a12a013c8e5468 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:17:38.040858 19012 raft_consensus.cc:686] T e101ffb35f7d4dff9f7a44ce6ed9650f P 1c02948bfdcc4179b0a12a013c8e5468 [term 1 LEADER]: Becoming Leader. State: Replica: 1c02948bfdcc4179b0a12a013c8e5468, State: Running, Role: LEADER
I20260924 04:17:38.041180 19012 consensus_queue.cc:237] T e101ffb35f7d4dff9f7a44ce6ed9650f P 1c02948bfdcc4179b0a12a013c8e5468 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179672a4c96e4581bae6eafee4a44f8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42171 } } peers { permanent_uuid: "1c02948bfdcc4179b0a12a013c8e5468" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34823 } } peers { permanent_uuid: "21a9624e47ec4741979c59b360a31630" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44533 } }
I20260924 04:17:38.044142 18529 catalog_manager.cc:5495] T e101ffb35f7d4dff9f7a44ce6ed9650f P 1c02948bfdcc4179b0a12a013c8e5468 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c02948bfdcc4179b0a12a013c8e5468 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "1c02948bfdcc4179b0a12a013c8e5468" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179672a4c96e4581bae6eafee4a44f8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c02948bfdcc4179b0a12a013c8e5468" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21a9624e47ec4741979c59b360a31630" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44533 } health_report { overall_health: UNKNOWN } } }
I20260924 04:17:38.056598 19004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:38.053548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38758 (local address 127.0.98.2:34823)
0924 04:17:38.053651 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:38.053654 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:38.053676 (+ 22us) server_negotiation.cc:407] Connection header received
0924 04:17:38.053786 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:38.053788 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:38.053837 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:38.053916 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:17:38.054090 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:38.054562 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:38.055355 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:38.055557 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:38.055625 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:38.055879 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:17:38.056256 (+ 377us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:17:38.056358 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:38.056412 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 04:17:38.056442 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 04:17:38.061913 18943 raft_consensus.cc:1261] T e101ffb35f7d4dff9f7a44ce6ed9650f P 21a9624e47ec4741979c59b360a31630 [term 1 FOLLOWER]: Refusing update from remote peer 1c02948bfdcc4179b0a12a013c8e5468: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:17:38.061913 18675 raft_consensus.cc:1261] T e101ffb35f7d4dff9f7a44ce6ed9650f P 179672a4c96e4581bae6eafee4a44f8f [term 1 FOLLOWER]: Refusing update from remote peer 1c02948bfdcc4179b0a12a013c8e5468: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:17:38.062573 19016 consensus_queue.cc:1035] T e101ffb35f7d4dff9f7a44ce6ed9650f P 1c02948bfdcc4179b0a12a013c8e5468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21a9624e47ec4741979c59b360a31630" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:17:38.062788 19012 consensus_queue.cc:1035] T e101ffb35f7d4dff9f7a44ce6ed9650f P 1c02948bfdcc4179b0a12a013c8e5468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "179672a4c96e4581bae6eafee4a44f8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:17:38.070013 19020 mvcc.cc:205] Tried to move back new op lower bound from 7332755284215173120 to 7332755284139020288. Current Snapshot: MvccSnapshot[applied={T|T < 7332755284215173120}]
I20260924 04:17:38.070297 19021 mvcc.cc:205] Tried to move back new op lower bound from 7332755284215173120 to 7332755284139020288. Current Snapshot: MvccSnapshot[applied={T|T < 7332755284215173120}]
I20260924 04:17:38.077663 18529 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:17:38.077814 18529 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:17:38.079830 18529 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1a67dac6771a4652bdabebde55b52ce9: Sending DeleteTablet for 3 replicas of tablet e101ffb35f7d4dff9f7a44ce6ed9650f
I20260924 04:17:38.080598 18789 tablet_service.cc:1514] Processing DeleteTablet for tablet e101ffb35f7d4dff9f7a44ce6ed9650f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:17:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38750
I20260924 04:17:38.080713 18655 tablet_service.cc:1514] Processing DeleteTablet for tablet e101ffb35f7d4dff9f7a44ce6ed9650f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:17:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45908
I20260924 04:17:38.081104 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 18592
I20260924 04:17:38.081143 18923 tablet_service.cc:1514] Processing DeleteTablet for tablet e101ffb35f7d4dff9f7a44ce6ed9650f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:17:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54114
I20260924 04:17:38.081213 19029 tablet_replica.cc:330] T e101ffb35f7d4dff9f7a44ce6ed9650f P 1c02948bfdcc4179b0a12a013c8e5468: stopping tablet replica
I20260924 04:17:38.082223 19029 raft_consensus.cc:2229] T e101ffb35f7d4dff9f7a44ce6ed9650f P 1c02948bfdcc4179b0a12a013c8e5468 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:17:38.082525 19029 raft_consensus.cc:2258] T e101ffb35f7d4dff9f7a44ce6ed9650f P 1c02948bfdcc4179b0a12a013c8e5468 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:17:38.083869 19029 ts_tablet_manager.cc:1905] T e101ffb35f7d4dff9f7a44ce6ed9650f P 1c02948bfdcc4179b0a12a013c8e5468: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:17:38.084142 19031 tablet_replica.cc:330] T e101ffb35f7d4dff9f7a44ce6ed9650f P 21a9624e47ec4741979c59b360a31630: stopping tablet replica
I20260924 04:17:38.084429 19031 raft_consensus.cc:2229] T e101ffb35f7d4dff9f7a44ce6ed9650f P 21a9624e47ec4741979c59b360a31630 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:17:38.084709 19031 raft_consensus.cc:2258] T e101ffb35f7d4dff9f7a44ce6ed9650f P 21a9624e47ec4741979c59b360a31630 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:17:38.086066 19031 ts_tablet_manager.cc:1905] T e101ffb35f7d4dff9f7a44ce6ed9650f P 21a9624e47ec4741979c59b360a31630: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:17:38.086627 19029 ts_tablet_manager.cc:1918] T e101ffb35f7d4dff9f7a44ce6ed9650f P 1c02948bfdcc4179b0a12a013c8e5468: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:17:38.086731 19029 log.cc:1196] T e101ffb35f7d4dff9f7a44ce6ed9650f P 1c02948bfdcc4179b0a12a013c8e5468: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/ts-1/wal/wals/e101ffb35f7d4dff9f7a44ce6ed9650f
I20260924 04:17:38.087026 19029 ts_tablet_manager.cc:1939] T e101ffb35f7d4dff9f7a44ce6ed9650f P 1c02948bfdcc4179b0a12a013c8e5468: Deleting consensus metadata
I20260924 04:17:38.088176 18516 catalog_manager.cc:4841] TS 1c02948bfdcc4179b0a12a013c8e5468 (127.0.98.2:34823): tablet e101ffb35f7d4dff9f7a44ce6ed9650f (table test-table [id=7a6bd98d541247bd847921c564c3b465]) successfully deleted
I20260924 04:17:38.089018 19031 ts_tablet_manager.cc:1918] T e101ffb35f7d4dff9f7a44ce6ed9650f P 21a9624e47ec4741979c59b360a31630: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:17:38.089102 19031 log.cc:1196] T e101ffb35f7d4dff9f7a44ce6ed9650f P 21a9624e47ec4741979c59b360a31630: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790223216265354-392-0/minicluster-data/ts-2/wal/wals/e101ffb35f7d4dff9f7a44ce6ed9650f
I20260924 04:17:38.089358 19031 ts_tablet_manager.cc:1939] T e101ffb35f7d4dff9f7a44ce6ed9650f P 21a9624e47ec4741979c59b360a31630: Deleting consensus metadata
I20260924 04:17:38.089821 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 18726
W20260924 04:17:38.089810 18517 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42171 (error 108)
I20260924 04:17:38.092000 18516 catalog_manager.cc:4841] TS 21a9624e47ec4741979c59b360a31630 (127.0.98.3:44533): tablet e101ffb35f7d4dff9f7a44ce6ed9650f (table test-table [id=7a6bd98d541247bd847921c564c3b465]) successfully deleted
I20260924 04:17:38.092577 19000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:38.092318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42171 (local address 127.0.0.1:45916)
0924 04:17:38.092436 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:38.092499 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 04:17:38.092729 18517 catalog_manager.cc:4584] TS 179672a4c96e4581bae6eafee4a44f8f (127.0.98.1:42171): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e101ffb35f7d4dff9f7a44ce6ed9650f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42171: connect: Connection refused (error 111)
I20260924 04:17:38.095079 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 18860
I20260924 04:17:38.099695 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 18498
2026-09-24T04:17:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2083 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-24T04:17:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:17:38Z Disabled control of system clock
I20260924 04:17:38.126327 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43517
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43517
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 04:17:38.232044 19036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:17:38.232322 19036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:17:38.232357 19036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:17:38.236644 19036 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:17:38.236707 19036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:17:38.236722 19036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:17:38.236738 19036 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:17:38.236752 19036 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:17:38.241578 19036 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43517
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:17:38.242638 19036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:17:38.244035 19036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:17:38.251974 19041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:17:38.252045 19036 server_base.cc:1034] running on GCE node
W20260924 04:17:38.251981 19042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:17:38.251993 19044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:17:38.252758 19036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:17:38.253834 19036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:17:38.254999 19036 hybrid_clock.cc:648] HybridClock initialized: now 1790223458254982 us; error 33 us; skew 500 ppm
I20260924 04:17:38.256795 19036 webserver.cc:460] Webserver started at http://127.0.98.62:35457/ using document root <none> and password file <none>
I20260924 04:17:38.257346 19036 fs_manager.cc:359] Metadata directory not provided
I20260924 04:17:38.257411 19036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:17:38.257609 19036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:17:38.259632 19036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/master-0/data/instance:
uuid: "eeb83a6d274141d9ae2183ece0539f3d"
format_stamp: "Formatted at 2026-09-24 04:17:38 on dist-test-slave-p401"
server_key: "f7fb4bf5f6022b4f42dd6f2be7bedc87"
server_key_iv: "b012c57dbd6dd4d9c967c011e93a3e7f"
server_key_version: "encryptionkey@0"
I20260924 04:17:38.260149 19036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/master-0/wal/instance:
uuid: "eeb83a6d274141d9ae2183ece0539f3d"
format_stamp: "Formatted at 2026-09-24 04:17:38 on dist-test-slave-p401"
server_key: "f7fb4bf5f6022b4f42dd6f2be7bedc87"
server_key_iv: "b012c57dbd6dd4d9c967c011e93a3e7f"
server_key_version: "encryptionkey@0"
I20260924 04:17:38.263609 19036 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:17:38.265888 19050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:38.267061 19036 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:17:38.267186 19036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/master-0/wal
uuid: "eeb83a6d274141d9ae2183ece0539f3d"
format_stamp: "Formatted at 2026-09-24 04:17:38 on dist-test-slave-p401"
server_key: "f7fb4bf5f6022b4f42dd6f2be7bedc87"
server_key_iv: "b012c57dbd6dd4d9c967c011e93a3e7f"
server_key_version: "encryptionkey@0"
I20260924 04:17:38.267318 19036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:17:38.285544 19036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:17:38.286269 19036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:17:38.286443 19036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:17:38.292903 19036 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43517
I20260924 04:17:38.292920 19102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43517 every 8 connection(s)
I20260924 04:17:38.293807 19036 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:17:38.296788 19103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:17:38.301877 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 19036
I20260924 04:17:38.301994 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/master-0/wal/instance
I20260924 04:17:38.302204 19103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eeb83a6d274141d9ae2183ece0539f3d: Bootstrap starting.
I20260924 04:17:38.302241 392 external_mini_cluster.cc:1428] Setting key ddd161dfdc28016568f74501cd94f6ad
I20260924 04:17:38.304839 19103 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eeb83a6d274141d9ae2183ece0539f3d: Neither blocks nor log segments found. Creating new log.
I20260924 04:17:38.305691 19103 log.cc:824] T 00000000000000000000000000000000 P eeb83a6d274141d9ae2183ece0539f3d: Log is configured to *not* fsync() on all Append() calls
I20260924 04:17:38.307852 19103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eeb83a6d274141d9ae2183ece0539f3d: No bootstrap required, opened a new log
I20260924 04:17:38.312069 19103 raft_consensus.cc:348] T 00000000000000000000000000000000 P eeb83a6d274141d9ae2183ece0539f3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeb83a6d274141d9ae2183ece0539f3d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43517 } }
I20260924 04:17:38.312342 19103 raft_consensus.cc:374] T 00000000000000000000000000000000 P eeb83a6d274141d9ae2183ece0539f3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:17:38.312384 19106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:38.303554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54690 (local address 127.0.98.62:43517)
0924 04:17:38.304232 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:38.304250 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:38.304290 (+ 40us) server_negotiation.cc:407] Connection header received
0924 04:17:38.305103 (+ 813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:38.305127 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:38.305633 (+ 506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:38.305997 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:17:38.306373 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:38.307767 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:38.308685 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:38.309374 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:38.309536 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:38.310236 (+ 700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:17:38.310242 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:17:38.310259 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:17:38.310764 (+ 505us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:17:38.310842 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:17:38.310872 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:17:38.310974 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:38.311055 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 04:17:38.311397 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":104,"threads_started":1}
I20260924 04:17:38.312403 19103 raft_consensus.cc:729] T 00000000000000000000000000000000 P eeb83a6d274141d9ae2183ece0539f3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeb83a6d274141d9ae2183ece0539f3d, State: Initialized, Role: FOLLOWER
I20260924 04:17:38.312820 19103 consensus_queue.cc:260] T 00000000000000000000000000000000 P eeb83a6d274141d9ae2183ece0539f3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeb83a6d274141d9ae2183ece0539f3d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43517 } }
I20260924 04:17:38.312942 19103 raft_consensus.cc:388] T 00000000000000000000000000000000 P eeb83a6d274141d9ae2183ece0539f3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:17:38.312997 19103 raft_consensus.cc:482] T 00000000000000000000000000000000 P eeb83a6d274141d9ae2183ece0539f3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:17:38.313090 19103 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eeb83a6d274141d9ae2183ece0539f3d [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:17:38.314157 19103 raft_consensus.cc:504] T 00000000000000000000000000000000 P eeb83a6d274141d9ae2183ece0539f3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeb83a6d274141d9ae2183ece0539f3d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43517 } }
I20260924 04:17:38.314499 19103 leader_election.cc:302] T 00000000000000000000000000000000 P eeb83a6d274141d9ae2183ece0539f3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eeb83a6d274141d9ae2183ece0539f3d; no voters:
I20260924 04:17:38.314831 19103 leader_election.cc:288] T 00000000000000000000000000000000 P eeb83a6d274141d9ae2183ece0539f3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:17:38.314915 19108 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eeb83a6d274141d9ae2183ece0539f3d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:17:38.315141 19108 raft_consensus.cc:686] T 00000000000000000000000000000000 P eeb83a6d274141d9ae2183ece0539f3d [term 1 LEADER]: Becoming Leader. State: Replica: eeb83a6d274141d9ae2183ece0539f3d, State: Running, Role: LEADER
I20260924 04:17:38.315477 19108 consensus_queue.cc:237] T 00000000000000000000000000000000 P eeb83a6d274141d9ae2183ece0539f3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeb83a6d274141d9ae2183ece0539f3d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43517 } }
I20260924 04:17:38.316045 19103 sys_catalog.cc:564] T 00000000000000000000000000000000 P eeb83a6d274141d9ae2183ece0539f3d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:17:38.316998 19110 sys_catalog.cc:455] T 00000000000000000000000000000000 P eeb83a6d274141d9ae2183ece0539f3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader eeb83a6d274141d9ae2183ece0539f3d. Latest consensus state: current_term: 1 leader_uuid: "eeb83a6d274141d9ae2183ece0539f3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeb83a6d274141d9ae2183ece0539f3d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43517 } } }
I20260924 04:17:38.317121 19110 sys_catalog.cc:458] T 00000000000000000000000000000000 P eeb83a6d274141d9ae2183ece0539f3d [sys.catalog]: This master's current role is: LEADER
I20260924 04:17:38.317333 19109 sys_catalog.cc:455] T 00000000000000000000000000000000 P eeb83a6d274141d9ae2183ece0539f3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eeb83a6d274141d9ae2183ece0539f3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeb83a6d274141d9ae2183ece0539f3d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43517 } } }
I20260924 04:17:38.317415 19109 sys_catalog.cc:458] T 00000000000000000000000000000000 P eeb83a6d274141d9ae2183ece0539f3d [sys.catalog]: This master's current role is: LEADER
I20260924 04:17:38.317461 19117 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:17:38.320294 19117 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:17:38.325206 19117 catalog_manager.cc:1347] Generated new cluster ID: a24c224b8389434a9d7895e4a4524da6
I20260924 04:17:38.325266 19117 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:17:38.335026 19117 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:17:38.336233 19117 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:17:38.344360 19117 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eeb83a6d274141d9ae2183ece0539f3d: Generated new TSK 0
I20260924 04:17:38.345155 19117 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:17:38.351120 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43517
--builtin_ntp_servers=127.0.98.20:43785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 04:17:38.458124 19127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:17:38.458384 19127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:17:38.458423 19127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:17:38.462340 19127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:17:38.462392 19127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:17:38.462446 19127 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:17:38.467090 19127 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:17:38.468116 19127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:17:38.469424 19127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:17:38.478753 19133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:17:38.478788 19135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:17:38.478770 19132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:17:38.479094 19127 server_base.cc:1034] running on GCE node
I20260924 04:17:38.479523 19127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:17:38.480183 19127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:17:38.481339 19127 hybrid_clock.cc:648] HybridClock initialized: now 1790223458481320 us; error 34 us; skew 500 ppm
I20260924 04:17:38.483135 19127 webserver.cc:460] Webserver started at http://127.0.98.1:36369/ using document root <none> and password file <none>
I20260924 04:17:38.483721 19127 fs_manager.cc:359] Metadata directory not provided
I20260924 04:17:38.483788 19127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:17:38.483970 19127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:17:38.485924 19127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/ts-0/data/instance:
uuid: "db6a4816d61741e6a4affcc787215f39"
format_stamp: "Formatted at 2026-09-24 04:17:38 on dist-test-slave-p401"
server_key: "a254c2873c1bacdac1a9ed24cc72410c"
server_key_iv: "caec66adfe165d0f2b8ee1a6293da155"
server_key_version: "encryptionkey@0"
I20260924 04:17:38.486467 19127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/ts-0/wal/instance:
uuid: "db6a4816d61741e6a4affcc787215f39"
format_stamp: "Formatted at 2026-09-24 04:17:38 on dist-test-slave-p401"
server_key: "a254c2873c1bacdac1a9ed24cc72410c"
server_key_iv: "caec66adfe165d0f2b8ee1a6293da155"
server_key_version: "encryptionkey@0"
I20260924 04:17:38.489899 19127 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260924 04:17:38.492110 19141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:38.493224 19127 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:17:38.493321 19127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/ts-0/wal
uuid: "db6a4816d61741e6a4affcc787215f39"
format_stamp: "Formatted at 2026-09-24 04:17:38 on dist-test-slave-p401"
server_key: "a254c2873c1bacdac1a9ed24cc72410c"
server_key_iv: "caec66adfe165d0f2b8ee1a6293da155"
server_key_version: "encryptionkey@0"
I20260924 04:17:38.493409 19127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:17:38.516502 19127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:17:38.517218 19127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:17:38.517386 19127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:17:38.517962 19127 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:17:38.518894 19127 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:17:38.518946 19127 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:38.519001 19127 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:17:38.519026 19127 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:38.526890 19127 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44099
I20260924 04:17:38.526916 19254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44099 every 8 connection(s)
I20260924 04:17:38.527751 19127 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:17:38.537009 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 19127
I20260924 04:17:38.537143 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:17:38.537521 392 external_mini_cluster.cc:1428] Setting key 887ee8ad163186f0eb83c70ee6586b26
I20260924 04:17:38.541826 19106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:38.529423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60091 (local address 127.0.98.62:43517)
0924 04:17:38.529542 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:38.529546 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:38.530530 (+ 984us) server_negotiation.cc:407] Connection header received
0924 04:17:38.531858 (+ 1328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:38.531861 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:38.531904 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:38.531964 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:17:38.532766 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:38.533345 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:38.534866 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:38.535362 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:38.535481 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:38.541332 (+ 5851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:17:38.541336 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:17:38.541339 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:17:38.541352 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:17:38.541376 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:17:38.541386 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:17:38.541459 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:38.541606 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 04:17:38.541680 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 04:17:38.541844 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43517
--builtin_ntp_servers=127.0.98.20:43785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 04:17:38.542708 19257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:38.529799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43517 (local address 127.0.98.1:60091)
0924 04:17:38.530397 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:38.530435 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 04:17:38.531643 (+ 1208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:17:38.532109 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:17:38.532115 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:17:38.532195 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:17:38.532649 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:38.532660 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:38.533442 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:38.533444 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:38.534684 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:38.534696 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:38.540113 (+ 5417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:38.540118 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:38.540335 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:38.541039 (+ 704us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:17:38.541041 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:17:38.541096 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:17:38.541187 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:17:38.541464 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:17:38.541471 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:17:38.541476 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 04:17:38.541586 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 04:17:38.541860 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":100,"threads_started":1}
I20260924 04:17:38.543862 19255 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43517
I20260924 04:17:38.544076 19255 heartbeater.cc:464] Registering TS with master...
I20260924 04:17:38.544544 19255 heartbeater.cc:511] Master 127.0.98.62:43517 requested a full tablet report, sending...
I20260924 04:17:38.546042 19067 ts_manager.cc:194] Registered new tserver with Master: db6a4816d61741e6a4affcc787215f39 (127.0.98.1:44099)
I20260924 04:17:38.547596 19067 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:60091
W20260924 04:17:38.647080 19258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:17:38.647343 19258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:17:38.647379 19258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:17:38.651222 19258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:17:38.651294 19258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:17:38.651346 19258 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:17:38.655898 19258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:17:38.656889 19258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:17:38.658237 19258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:17:38.666994 19264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:17:38.667037 19263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:17:38.667088 19258 server_base.cc:1034] running on GCE node
W20260924 04:17:38.667008 19266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:17:38.667696 19258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:17:38.668242 19258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:17:38.669471 19258 hybrid_clock.cc:648] HybridClock initialized: now 1790223458669451 us; error 28 us; skew 500 ppm
I20260924 04:17:38.671267 19258 webserver.cc:460] Webserver started at http://127.0.98.2:35119/ using document root <none> and password file <none>
I20260924 04:17:38.671804 19258 fs_manager.cc:359] Metadata directory not provided
I20260924 04:17:38.671866 19258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:17:38.672047 19258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:17:38.673956 19258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/ts-1/data/instance:
uuid: "9987807937db449a8d2cb8078493bc25"
format_stamp: "Formatted at 2026-09-24 04:17:38 on dist-test-slave-p401"
server_key: "0387de19615edb5a849bf2330b94abdd"
server_key_iv: "53483b6253af79c5a65e100e201dbe50"
server_key_version: "encryptionkey@0"
I20260924 04:17:38.674458 19258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9987807937db449a8d2cb8078493bc25"
format_stamp: "Formatted at 2026-09-24 04:17:38 on dist-test-slave-p401"
server_key: "0387de19615edb5a849bf2330b94abdd"
server_key_iv: "53483b6253af79c5a65e100e201dbe50"
server_key_version: "encryptionkey@0"
I20260924 04:17:38.677773 19258 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.001s
I20260924 04:17:38.680051 19272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:38.681166 19258 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:17:38.681258 19258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/ts-1/wal
uuid: "9987807937db449a8d2cb8078493bc25"
format_stamp: "Formatted at 2026-09-24 04:17:38 on dist-test-slave-p401"
server_key: "0387de19615edb5a849bf2330b94abdd"
server_key_iv: "53483b6253af79c5a65e100e201dbe50"
server_key_version: "encryptionkey@0"
I20260924 04:17:38.681352 19258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:17:38.701220 19258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:17:38.701869 19258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:17:38.702031 19258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:17:38.702584 19258 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:17:38.703502 19258 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:17:38.703548 19258 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:38.703593 19258 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:17:38.703621 19258 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:38.711440 19258 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44855
I20260924 04:17:38.711474 19385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44855 every 8 connection(s)
I20260924 04:17:38.712298 19258 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:17:38.717514 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 19258
I20260924 04:17:38.717626 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:17:38.717875 392 external_mini_cluster.cc:1428] Setting key 29adf4334b74f170aeb1d81921be81f7
I20260924 04:17:38.719750 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43517
--builtin_ntp_servers=127.0.98.20:43785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 04:17:38.721268 19106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:38.713915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36801 (local address 127.0.98.62:43517)
0924 04:17:38.714030 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:38.714034 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:38.715058 (+ 1024us) server_negotiation.cc:407] Connection header received
0924 04:17:38.716423 (+ 1365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:38.716427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:38.716484 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:38.716555 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:17:38.717414 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:38.718056 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:38.719372 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:38.719584 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:38.719674 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:38.720707 (+ 1033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:17:38.720711 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:17:38.720714 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:17:38.720747 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:17:38.720773 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:17:38.720783 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:17:38.720842 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:38.721046 (+ 204us) server_negotiation.cc:299] Negotiation successful
0924 04:17:38.721126 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 04:17:38.722340 19388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:38.714304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43517 (local address 127.0.98.2:36801)
0924 04:17:38.714912 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:38.714950 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 04:17:38.716213 (+ 1263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:17:38.716726 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:17:38.716733 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:17:38.716816 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:17:38.717281 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:38.717294 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:38.718180 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:38.718183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:38.719264 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:38.719274 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:38.719707 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:38.719711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:38.719881 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:38.720459 (+ 578us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:17:38.720461 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:17:38.720500 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:17:38.720564 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:17:38.720879 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:17:38.720888 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:17:38.720894 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 04:17:38.721023 (+ 129us) client_negotiation.cc:240] Negotiation successful
0924 04:17:38.721319 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":101,"threads_started":1}
I20260924 04:17:38.724408 19386 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43517
I20260924 04:17:38.724628 19386 heartbeater.cc:464] Registering TS with master...
I20260924 04:17:38.725112 19386 heartbeater.cc:511] Master 127.0.98.62:43517 requested a full tablet report, sending...
I20260924 04:17:38.726073 19067 ts_manager.cc:194] Registered new tserver with Master: 9987807937db449a8d2cb8078493bc25 (127.0.98.2:44855)
I20260924 04:17:38.726862 19067 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:36801
W20260924 04:17:38.827378 19389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:17:38.827615 19389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:17:38.827658 19389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:17:38.831559 19389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:17:38.831609 19389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:17:38.831672 19389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:17:38.836302 19389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:17:38.837261 19389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:17:38.838538 19389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:17:38.847477 19395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:17:38.847517 19397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:17:38.847532 19394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:17:38.847512 19389 server_base.cc:1034] running on GCE node
I20260924 04:17:38.848079 19389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:17:38.848646 19389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:17:38.849800 19389 hybrid_clock.cc:648] HybridClock initialized: now 1790223458849772 us; error 38 us; skew 500 ppm
I20260924 04:17:38.851472 19389 webserver.cc:460] Webserver started at http://127.0.98.3:46555/ using document root <none> and password file <none>
I20260924 04:17:38.851999 19389 fs_manager.cc:359] Metadata directory not provided
I20260924 04:17:38.852057 19389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:17:38.852259 19389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:17:38.854151 19389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/ts-2/data/instance:
uuid: "83ff1653f2ad4794a097905d3e60e21e"
format_stamp: "Formatted at 2026-09-24 04:17:38 on dist-test-slave-p401"
server_key: "8f6444b60962d5be378adf27fce757c1"
server_key_iv: "598e9a8378a12af9defaa421343ae3f3"
server_key_version: "encryptionkey@0"
I20260924 04:17:38.854652 19389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/ts-2/wal/instance:
uuid: "83ff1653f2ad4794a097905d3e60e21e"
format_stamp: "Formatted at 2026-09-24 04:17:38 on dist-test-slave-p401"
server_key: "8f6444b60962d5be378adf27fce757c1"
server_key_iv: "598e9a8378a12af9defaa421343ae3f3"
server_key_version: "encryptionkey@0"
I20260924 04:17:38.858016 19389 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.001s
I20260924 04:17:38.860198 19403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:38.861284 19389 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:17:38.861371 19389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/ts-2/wal
uuid: "83ff1653f2ad4794a097905d3e60e21e"
format_stamp: "Formatted at 2026-09-24 04:17:38 on dist-test-slave-p401"
server_key: "8f6444b60962d5be378adf27fce757c1"
server_key_iv: "598e9a8378a12af9defaa421343ae3f3"
server_key_version: "encryptionkey@0"
I20260924 04:17:38.861462 19389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:17:38.873757 19389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:17:38.874318 19389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:17:38.874462 19389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:17:38.874963 19389 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:17:38.875909 19389 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:17:38.875953 19389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:38.875993 19389 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:17:38.876020 19389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:38.883589 19389 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45559
I20260924 04:17:38.883634 19516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45559 every 8 connection(s)
I20260924 04:17:38.884415 19389 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:17:38.885270 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 19389
I20260924 04:17:38.885371 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:17:38.885654 392 external_mini_cluster.cc:1428] Setting key a54e6e9c2348ff941da0f50dd6cd7deb
I20260924 04:17:38.892647 19106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:38.886200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51151 (local address 127.0.98.62:43517)
0924 04:17:38.886339 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:38.886344 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:38.887376 (+ 1032us) server_negotiation.cc:407] Connection header received
0924 04:17:38.888758 (+ 1382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:38.888761 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:38.888803 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:38.888857 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:17:38.889613 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:38.890116 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:38.891124 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:38.891377 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:38.891448 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:38.892311 (+ 863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:17:38.892314 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:17:38.892316 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:17:38.892326 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:17:38.892344 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:17:38.892351 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:17:38.892401 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:38.892492 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 04:17:38.892539 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 04:17:38.893411 19519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:38.886619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43517 (local address 127.0.98.3:51151)
0924 04:17:38.887221 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:38.887291 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0924 04:17:38.888579 (+ 1288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:17:38.888984 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:17:38.888989 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:17:38.889066 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:17:38.889515 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:38.889526 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:38.890208 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:38.890210 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:38.891042 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:38.891050 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:38.891479 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:38.891482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:38.891611 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:38.892094 (+ 483us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:17:38.892095 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:17:38.892131 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:17:38.892193 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:17:38.892402 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:17:38.892405 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:17:38.892409 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 04:17:38.892484 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 04:17:38.892672 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":100,"threads_started":1}
I20260924 04:17:38.894346 19517 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43517
I20260924 04:17:38.894523 19517 heartbeater.cc:464] Registering TS with master...
I20260924 04:17:38.894948 19517 heartbeater.cc:511] Master 127.0.98.62:43517 requested a full tablet report, sending...
I20260924 04:17:38.895836 19067 ts_manager.cc:194] Registered new tserver with Master: 83ff1653f2ad4794a097905d3e60e21e (127.0.98.3:45559)
I20260924 04:17:38.896396 19067 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:51151
I20260924 04:17:38.900092 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:17:38.904976 19106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:38.901764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54696 (local address 127.0.98.62:43517)
0924 04:17:38.901879 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:38.901883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:38.902033 (+ 150us) server_negotiation.cc:407] Connection header received
0924 04:17:38.902126 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:38.902128 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:38.902164 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:38.902218 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:17:38.902417 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:38.902892 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:38.903664 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:38.903867 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:38.903934 (+ 67us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:38.904583 (+ 649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:17:38.904586 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:17:38.904588 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:17:38.904598 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:17:38.904614 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:17:38.904621 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:17:38.904719 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:38.904809 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 04:17:38.904843 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 04:17:38.907539 19067 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54696:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:17:38.910426 19067 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:17:38.923825 19528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:38.918164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44099 (local address 127.0.0.1:33134)
0924 04:17:38.918476 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:38.918528 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 04:17:38.918649 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:17:38.919696 (+ 1047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:17:38.919701 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:17:38.919722 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:17:38.919863 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:38.919871 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:38.921291 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:38.921294 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:38.922384 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:38.922391 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:38.923376 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:38.923380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:38.923482 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:38.923518 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 04:17:38.923620 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 04:17:38.923715 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":1,"thread_start_us":66,"threads_started":1}
I20260924 04:17:38.923835 19530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:38.918475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33134 (local address 127.0.98.1:44099)
0924 04:17:38.919184 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:38.919197 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:38.919220 (+ 23us) server_negotiation.cc:407] Connection header received
0924 04:17:38.919322 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:38.919327 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:38.919539 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:38.919708 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:17:38.920044 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:38.921171 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:38.922492 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:38.923275 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:38.923371 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:38.923549 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:38.923641 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 04:17:38.923716 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":115,"threads_started":1}
I20260924 04:17:38.924301 19529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:38.918220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45559 (local address 127.0.0.1:55540)
0924 04:17:38.918834 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:38.918846 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:17:38.918926 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:17:38.919911 (+ 985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:17:38.919913 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:17:38.919930 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:17:38.920017 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:38.920023 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:38.921036 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:38.921039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:38.922327 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:38.922337 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:38.923983 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:38.923987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:38.924066 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:38.924087 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 04:17:38.924147 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 04:17:38.924208 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":78,"threads_started":1}
I20260924 04:17:38.924543 19532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:38.918952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55540 (local address 127.0.98.3:45559)
0924 04:17:38.919443 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:38.919453 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:38.919475 (+ 22us) server_negotiation.cc:407] Connection header received
0924 04:17:38.919546 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:38.919551 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:38.919736 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:38.919940 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:17:38.920132 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:38.920949 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:38.922463 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:38.923268 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:38.923371 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:38.923549 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:38.924190 (+ 641us) server_negotiation.cc:299] Negotiation successful
0924 04:17:38.924288 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":157,"threads_started":1}
I20260924 04:17:38.924782 19531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:38.918847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44855 (local address 127.0.0.1:41454)
0924 04:17:38.919910 (+ 1063us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:38.919922 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:17:38.919998 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:17:38.920460 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:17:38.920463 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:17:38.920480 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:17:38.920555 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:38.920560 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:38.921687 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:38.921690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:38.922630 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:38.922640 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:38.924430 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:38.924433 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:38.924513 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:38.924533 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 04:17:38.924597 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 04:17:38.924660 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1006,"thread_start_us":69,"threads_started":1}
I20260924 04:17:38.925083 19533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:38.919135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41454 (local address 127.0.98.2:44855)
0924 04:17:38.919514 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:38.919523 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:38.920078 (+ 555us) server_negotiation.cc:407] Connection header received
0924 04:17:38.920151 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:38.920156 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:38.920320 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:38.920462 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:17:38.920663 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:38.921593 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:38.922753 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:38.923276 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:38.923376 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:38.923550 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:38.924749 (+ 1199us) server_negotiation.cc:299] Negotiation successful
0924 04:17:38.924969 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":128,"threads_started":1}
I20260924 04:17:38.926736 19189 tablet_service.cc:1467] Processing CreateTablet for tablet 01c440b9aa6c49c587994c2c3298a31f (DEFAULT_TABLE table=test-table [id=9faf8da19a454fac856881c3e3524dc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:17:38.926839 19451 tablet_service.cc:1467] Processing CreateTablet for tablet 01c440b9aa6c49c587994c2c3298a31f (DEFAULT_TABLE table=test-table [id=9faf8da19a454fac856881c3e3524dc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:17:38.927644 19451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01c440b9aa6c49c587994c2c3298a31f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:17:38.927644 19189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01c440b9aa6c49c587994c2c3298a31f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:17:38.932003 19320 tablet_service.cc:1467] Processing CreateTablet for tablet 01c440b9aa6c49c587994c2c3298a31f (DEFAULT_TABLE table=test-table [id=9faf8da19a454fac856881c3e3524dc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:17:38.933002 19320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01c440b9aa6c49c587994c2c3298a31f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:17:38.933354 19534 tablet_bootstrap.cc:492] T 01c440b9aa6c49c587994c2c3298a31f P 83ff1653f2ad4794a097905d3e60e21e: Bootstrap starting.
I20260924 04:17:38.934741 19535 tablet_bootstrap.cc:492] T 01c440b9aa6c49c587994c2c3298a31f P db6a4816d61741e6a4affcc787215f39: Bootstrap starting.
I20260924 04:17:38.935411 19534 tablet_bootstrap.cc:654] T 01c440b9aa6c49c587994c2c3298a31f P 83ff1653f2ad4794a097905d3e60e21e: Neither blocks nor log segments found. Creating new log.
I20260924 04:17:38.936074 19534 log.cc:824] T 01c440b9aa6c49c587994c2c3298a31f P 83ff1653f2ad4794a097905d3e60e21e: Log is configured to *not* fsync() on all Append() calls
I20260924 04:17:38.936770 19535 tablet_bootstrap.cc:654] T 01c440b9aa6c49c587994c2c3298a31f P db6a4816d61741e6a4affcc787215f39: Neither blocks nor log segments found. Creating new log.
I20260924 04:17:38.937561 19535 log.cc:824] T 01c440b9aa6c49c587994c2c3298a31f P db6a4816d61741e6a4affcc787215f39: Log is configured to *not* fsync() on all Append() calls
I20260924 04:17:38.937703 19534 tablet_bootstrap.cc:492] T 01c440b9aa6c49c587994c2c3298a31f P 83ff1653f2ad4794a097905d3e60e21e: No bootstrap required, opened a new log
I20260924 04:17:38.937872 19534 ts_tablet_manager.cc:1397] T 01c440b9aa6c49c587994c2c3298a31f P 83ff1653f2ad4794a097905d3e60e21e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 04:17:38.939546 19535 tablet_bootstrap.cc:492] T 01c440b9aa6c49c587994c2c3298a31f P db6a4816d61741e6a4affcc787215f39: No bootstrap required, opened a new log
I20260924 04:17:38.939764 19535 ts_tablet_manager.cc:1397] T 01c440b9aa6c49c587994c2c3298a31f P db6a4816d61741e6a4affcc787215f39: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260924 04:17:38.940559 19538 tablet_bootstrap.cc:492] T 01c440b9aa6c49c587994c2c3298a31f P 9987807937db449a8d2cb8078493bc25: Bootstrap starting.
I20260924 04:17:38.941381 19534 raft_consensus.cc:348] T 01c440b9aa6c49c587994c2c3298a31f P 83ff1653f2ad4794a097905d3e60e21e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6a4816d61741e6a4affcc787215f39" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44099 } } peers { permanent_uuid: "83ff1653f2ad4794a097905d3e60e21e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45559 } } peers { permanent_uuid: "9987807937db449a8d2cb8078493bc25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44855 } }
I20260924 04:17:38.941576 19534 raft_consensus.cc:374] T 01c440b9aa6c49c587994c2c3298a31f P 83ff1653f2ad4794a097905d3e60e21e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:17:38.941633 19534 raft_consensus.cc:729] T 01c440b9aa6c49c587994c2c3298a31f P 83ff1653f2ad4794a097905d3e60e21e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83ff1653f2ad4794a097905d3e60e21e, State: Initialized, Role: FOLLOWER
I20260924 04:17:38.942034 19534 consensus_queue.cc:260] T 01c440b9aa6c49c587994c2c3298a31f P 83ff1653f2ad4794a097905d3e60e21e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6a4816d61741e6a4affcc787215f39" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44099 } } peers { permanent_uuid: "83ff1653f2ad4794a097905d3e60e21e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45559 } } peers { permanent_uuid: "9987807937db449a8d2cb8078493bc25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44855 } }
I20260924 04:17:38.942543 19538 tablet_bootstrap.cc:654] T 01c440b9aa6c49c587994c2c3298a31f P 9987807937db449a8d2cb8078493bc25: Neither blocks nor log segments found. Creating new log.
I20260924 04:17:38.942592 19517 heartbeater.cc:503] Master 127.0.98.62:43517 was elected leader, sending a full tablet report...
I20260924 04:17:38.942926 19534 ts_tablet_manager.cc:1428] T 01c440b9aa6c49c587994c2c3298a31f P 83ff1653f2ad4794a097905d3e60e21e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 04:17:38.943338 19538 log.cc:824] T 01c440b9aa6c49c587994c2c3298a31f P 9987807937db449a8d2cb8078493bc25: Log is configured to *not* fsync() on all Append() calls
I20260924 04:17:38.943724 19535 raft_consensus.cc:348] T 01c440b9aa6c49c587994c2c3298a31f P db6a4816d61741e6a4affcc787215f39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6a4816d61741e6a4affcc787215f39" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44099 } } peers { permanent_uuid: "83ff1653f2ad4794a097905d3e60e21e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45559 } } peers { permanent_uuid: "9987807937db449a8d2cb8078493bc25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44855 } }
I20260924 04:17:38.943914 19535 raft_consensus.cc:374] T 01c440b9aa6c49c587994c2c3298a31f P db6a4816d61741e6a4affcc787215f39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:17:38.943948 19535 raft_consensus.cc:729] T 01c440b9aa6c49c587994c2c3298a31f P db6a4816d61741e6a4affcc787215f39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db6a4816d61741e6a4affcc787215f39, State: Initialized, Role: FOLLOWER
I20260924 04:17:38.944291 19535 consensus_queue.cc:260] T 01c440b9aa6c49c587994c2c3298a31f P db6a4816d61741e6a4affcc787215f39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6a4816d61741e6a4affcc787215f39" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44099 } } peers { permanent_uuid: "83ff1653f2ad4794a097905d3e60e21e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45559 } } peers { permanent_uuid: "9987807937db449a8d2cb8078493bc25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44855 } }
I20260924 04:17:38.944851 19255 heartbeater.cc:503] Master 127.0.98.62:43517 was elected leader, sending a full tablet report...
I20260924 04:17:38.945062 19535 ts_tablet_manager.cc:1428] T 01c440b9aa6c49c587994c2c3298a31f P db6a4816d61741e6a4affcc787215f39: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260924 04:17:38.946807 19538 tablet_bootstrap.cc:492] T 01c440b9aa6c49c587994c2c3298a31f P 9987807937db449a8d2cb8078493bc25: No bootstrap required, opened a new log
I20260924 04:17:38.946983 19538 ts_tablet_manager.cc:1397] T 01c440b9aa6c49c587994c2c3298a31f P 9987807937db449a8d2cb8078493bc25: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260924 04:17:38.950130 19538 raft_consensus.cc:348] T 01c440b9aa6c49c587994c2c3298a31f P 9987807937db449a8d2cb8078493bc25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6a4816d61741e6a4affcc787215f39" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44099 } } peers { permanent_uuid: "83ff1653f2ad4794a097905d3e60e21e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45559 } } peers { permanent_uuid: "9987807937db449a8d2cb8078493bc25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44855 } }
I20260924 04:17:38.950320 19538 raft_consensus.cc:374] T 01c440b9aa6c49c587994c2c3298a31f P 9987807937db449a8d2cb8078493bc25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:17:38.950357 19538 raft_consensus.cc:729] T 01c440b9aa6c49c587994c2c3298a31f P 9987807937db449a8d2cb8078493bc25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9987807937db449a8d2cb8078493bc25, State: Initialized, Role: FOLLOWER
I20260924 04:17:38.950655 19538 consensus_queue.cc:260] T 01c440b9aa6c49c587994c2c3298a31f P 9987807937db449a8d2cb8078493bc25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6a4816d61741e6a4affcc787215f39" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44099 } } peers { permanent_uuid: "83ff1653f2ad4794a097905d3e60e21e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45559 } } peers { permanent_uuid: "9987807937db449a8d2cb8078493bc25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44855 } }
I20260924 04:17:38.951164 19386 heartbeater.cc:503] Master 127.0.98.62:43517 was elected leader, sending a full tablet report...
I20260924 04:17:38.951402 19538 ts_tablet_manager.cc:1428] T 01c440b9aa6c49c587994c2c3298a31f P 9987807937db449a8d2cb8078493bc25: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
W20260924 04:17:38.963119 19387 tablet.cc:2369] T 01c440b9aa6c49c587994c2c3298a31f P 9987807937db449a8d2cb8078493bc25: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:17:38.967217 19542 raft_consensus.cc:482] T 01c440b9aa6c49c587994c2c3298a31f P 9987807937db449a8d2cb8078493bc25 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:17:38.967425 19542 raft_consensus.cc:504] T 01c440b9aa6c49c587994c2c3298a31f P 9987807937db449a8d2cb8078493bc25 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6a4816d61741e6a4affcc787215f39" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44099 } } peers { permanent_uuid: "83ff1653f2ad4794a097905d3e60e21e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45559 } } peers { permanent_uuid: "9987807937db449a8d2cb8078493bc25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44855 } }
I20260924 04:17:38.968506 19542 leader_election.cc:288] T 01c440b9aa6c49c587994c2c3298a31f P 9987807937db449a8d2cb8078493bc25 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db6a4816d61741e6a4affcc787215f39 (127.0.98.1:44099), 83ff1653f2ad4794a097905d3e60e21e (127.0.98.3:45559)
I20260924 04:17:38.972527 19388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:38.968766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45559 (local address 127.0.98.2:52759)
0924 04:17:38.968946 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:38.968960 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:17:38.969048 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:17:38.969350 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:17:38.969353 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:17:38.969367 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:17:38.969470 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:38.969476 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:38.970433 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:38.970435 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:38.971301 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:38.971310 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:38.972116 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:38.972120 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:38.972241 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:38.972267 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 04:17:38.972349 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 04:17:38.972414 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260924 04:17:38.972602 19532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:38.968865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52759 (local address 127.0.98.3:45559)
0924 04:17:38.968990 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:38.968994 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:38.969008 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:17:38.969192 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:38.969196 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:38.969247 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:38.969331 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:17:38.969581 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:38.970331 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:38.971414 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:38.972003 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:38.972092 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:38.972142 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:38.972373 (+ 231us) server_negotiation.cc:299] Negotiation successful
0924 04:17:38.972450 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 04:17:38.972889 19530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:38.968986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56507 (local address 127.0.98.1:44099)
0924 04:17:38.969169 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:38.969174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:38.969281 (+ 107us) server_negotiation.cc:407] Connection header received
0924 04:17:38.969370 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:38.969373 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:38.969423 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:38.969494 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:17:38.969728 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:38.970470 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:38.971829 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:38.972284 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:38.972374 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:38.972425 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:38.972685 (+ 260us) server_negotiation.cc:299] Negotiation successful
0924 04:17:38.972769 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260924 04:17:38.973210 19543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:38.968786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44099 (local address 127.0.98.2:56507)
0924 04:17:38.969205 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:38.969216 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:17:38.969292 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:17:38.969514 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:17:38.969516 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:17:38.969532 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:17:38.969631 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:38.969635 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:38.970628 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:38.970631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:38.971700 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:38.971712 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:38.972408 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:38.972412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:38.972513 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:38.972542 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 04:17:38.973042 (+ 500us) client_negotiation.cc:240] Negotiation successful
0924 04:17:38.973104 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"mutex_wait_us":1,"thread_start_us":138,"threads_started":1}
I20260924 04:17:38.973227 19471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01c440b9aa6c49c587994c2c3298a31f" candidate_uuid: "9987807937db449a8d2cb8078493bc25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83ff1653f2ad4794a097905d3e60e21e" is_pre_election: true
I20260924 04:17:38.973495 19471 raft_consensus.cc:2454] T 01c440b9aa6c49c587994c2c3298a31f P 83ff1653f2ad4794a097905d3e60e21e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9987807937db449a8d2cb8078493bc25 in term 0.
I20260924 04:17:38.973834 19209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01c440b9aa6c49c587994c2c3298a31f" candidate_uuid: "9987807937db449a8d2cb8078493bc25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6a4816d61741e6a4affcc787215f39" is_pre_election: true
I20260924 04:17:38.973976 19275 leader_election.cc:302] T 01c440b9aa6c49c587994c2c3298a31f P 9987807937db449a8d2cb8078493bc25 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83ff1653f2ad4794a097905d3e60e21e, 9987807937db449a8d2cb8078493bc25; no voters:
I20260924 04:17:38.974102 19209 raft_consensus.cc:2454] T 01c440b9aa6c49c587994c2c3298a31f P db6a4816d61741e6a4affcc787215f39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9987807937db449a8d2cb8078493bc25 in term 0.
I20260924 04:17:38.974231 19542 raft_consensus.cc:2781] T 01c440b9aa6c49c587994c2c3298a31f P 9987807937db449a8d2cb8078493bc25 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:17:38.974304 19542 raft_consensus.cc:482] T 01c440b9aa6c49c587994c2c3298a31f P 9987807937db449a8d2cb8078493bc25 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:17:38.974347 19542 raft_consensus.cc:3037] T 01c440b9aa6c49c587994c2c3298a31f P 9987807937db449a8d2cb8078493bc25 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:17:38.975387 19542 raft_consensus.cc:504] T 01c440b9aa6c49c587994c2c3298a31f P 9987807937db449a8d2cb8078493bc25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6a4816d61741e6a4affcc787215f39" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44099 } } peers { permanent_uuid: "83ff1653f2ad4794a097905d3e60e21e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45559 } } peers { permanent_uuid: "9987807937db449a8d2cb8078493bc25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44855 } }
I20260924 04:17:38.975706 19542 leader_election.cc:288] T 01c440b9aa6c49c587994c2c3298a31f P 9987807937db449a8d2cb8078493bc25 [CANDIDATE]: Term 1 election: Requested vote from peers db6a4816d61741e6a4affcc787215f39 (127.0.98.1:44099), 83ff1653f2ad4794a097905d3e60e21e (127.0.98.3:45559)
I20260924 04:17:38.976016 19209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01c440b9aa6c49c587994c2c3298a31f" candidate_uuid: "9987807937db449a8d2cb8078493bc25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6a4816d61741e6a4affcc787215f39"
I20260924 04:17:38.976012 19471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01c440b9aa6c49c587994c2c3298a31f" candidate_uuid: "9987807937db449a8d2cb8078493bc25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83ff1653f2ad4794a097905d3e60e21e"
I20260924 04:17:38.976155 19471 raft_consensus.cc:3037] T 01c440b9aa6c49c587994c2c3298a31f P 83ff1653f2ad4794a097905d3e60e21e [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:17:38.976135 19209 raft_consensus.cc:3037] T 01c440b9aa6c49c587994c2c3298a31f P db6a4816d61741e6a4affcc787215f39 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:17:38.977367 19471 raft_consensus.cc:2454] T 01c440b9aa6c49c587994c2c3298a31f P 83ff1653f2ad4794a097905d3e60e21e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9987807937db449a8d2cb8078493bc25 in term 1.
I20260924 04:17:38.977367 19209 raft_consensus.cc:2454] T 01c440b9aa6c49c587994c2c3298a31f P db6a4816d61741e6a4affcc787215f39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9987807937db449a8d2cb8078493bc25 in term 1.
I20260924 04:17:38.977631 19275 leader_election.cc:302] T 01c440b9aa6c49c587994c2c3298a31f P 9987807937db449a8d2cb8078493bc25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83ff1653f2ad4794a097905d3e60e21e, 9987807937db449a8d2cb8078493bc25; no voters:
I20260924 04:17:38.977788 19542 raft_consensus.cc:2781] T 01c440b9aa6c49c587994c2c3298a31f P 9987807937db449a8d2cb8078493bc25 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:17:38.977998 19542 raft_consensus.cc:686] T 01c440b9aa6c49c587994c2c3298a31f P 9987807937db449a8d2cb8078493bc25 [term 1 LEADER]: Becoming Leader. State: Replica: 9987807937db449a8d2cb8078493bc25, State: Running, Role: LEADER
I20260924 04:17:38.978363 19542 consensus_queue.cc:237] T 01c440b9aa6c49c587994c2c3298a31f P 9987807937db449a8d2cb8078493bc25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6a4816d61741e6a4affcc787215f39" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44099 } } peers { permanent_uuid: "83ff1653f2ad4794a097905d3e60e21e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45559 } } peers { permanent_uuid: "9987807937db449a8d2cb8078493bc25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44855 } }
I20260924 04:17:38.981487 19067 catalog_manager.cc:5495] T 01c440b9aa6c49c587994c2c3298a31f P 9987807937db449a8d2cb8078493bc25 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9987807937db449a8d2cb8078493bc25 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "9987807937db449a8d2cb8078493bc25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6a4816d61741e6a4affcc787215f39" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83ff1653f2ad4794a097905d3e60e21e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9987807937db449a8d2cb8078493bc25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44855 } health_report { overall_health: HEALTHY } } }
I20260924 04:17:39.002414 19533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:38.999058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41468 (local address 127.0.98.2:44855)
0924 04:17:38.999185 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:38.999190 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:38.999203 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:17:38.999339 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:38.999342 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:38.999393 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:38.999479 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:17:38.999710 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:39.000213 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:39.001039 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:39.001251 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:39.001334 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:39.001630 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:17:39.002019 (+ 389us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:17:39.002144 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:39.002242 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 04:17:39.002277 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 04:17:39.007791 19209 raft_consensus.cc:1261] T 01c440b9aa6c49c587994c2c3298a31f P db6a4816d61741e6a4affcc787215f39 [term 1 FOLLOWER]: Refusing update from remote peer 9987807937db449a8d2cb8078493bc25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:17:39.007791 19471 raft_consensus.cc:1261] T 01c440b9aa6c49c587994c2c3298a31f P 83ff1653f2ad4794a097905d3e60e21e [term 1 FOLLOWER]: Refusing update from remote peer 9987807937db449a8d2cb8078493bc25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:17:39.008472 19542 consensus_queue.cc:1035] T 01c440b9aa6c49c587994c2c3298a31f P 9987807937db449a8d2cb8078493bc25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6a4816d61741e6a4affcc787215f39" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:17:39.008663 19544 consensus_queue.cc:1035] T 01c440b9aa6c49c587994c2c3298a31f P 9987807937db449a8d2cb8078493bc25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83ff1653f2ad4794a097905d3e60e21e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:17:39.016779 19549 mvcc.cc:205] Tried to move back new op lower bound from 7332755288089178112 to 7332755287978024960. Current Snapshot: MvccSnapshot[applied={T|T < 7332755288089178112}]
I20260924 04:17:39.016866 19548 mvcc.cc:205] Tried to move back new op lower bound from 7332755288089178112 to 7332755287978024960. Current Snapshot: MvccSnapshot[applied={T|T < 7332755288089178112}]
I20260924 04:17:39.026297 19065 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:17:39.026456 19065 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:17:39.028741 19065 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eeb83a6d274141d9ae2183ece0539f3d: Sending DeleteTablet for 3 replicas of tablet 01c440b9aa6c49c587994c2c3298a31f
I20260924 04:17:39.029565 19189 tablet_service.cc:1514] Processing DeleteTablet for tablet 01c440b9aa6c49c587994c2c3298a31f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:17:39 UTC) from {username='slave'} at 127.0.0.1:33134
I20260924 04:17:39.030119 19320 tablet_service.cc:1514] Processing DeleteTablet for tablet 01c440b9aa6c49c587994c2c3298a31f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:17:39 UTC) from {username='slave'} at 127.0.0.1:41454
I20260924 04:17:39.030148 19451 tablet_service.cc:1514] Processing DeleteTablet for tablet 01c440b9aa6c49c587994c2c3298a31f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:17:39 UTC) from {username='slave'} at 127.0.0.1:55540
I20260924 04:17:39.030328 19558 tablet_replica.cc:330] T 01c440b9aa6c49c587994c2c3298a31f P db6a4816d61741e6a4affcc787215f39: stopping tablet replica
I20260924 04:17:39.030572 19559 tablet_replica.cc:330] T 01c440b9aa6c49c587994c2c3298a31f P 9987807937db449a8d2cb8078493bc25: stopping tablet replica
I20260924 04:17:39.030661 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 19127
I20260924 04:17:39.030732 19558 raft_consensus.cc:2229] T 01c440b9aa6c49c587994c2c3298a31f P db6a4816d61741e6a4affcc787215f39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:17:39.030850 19559 raft_consensus.cc:2229] T 01c440b9aa6c49c587994c2c3298a31f P 9987807937db449a8d2cb8078493bc25 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:17:39.031255 19559 raft_consensus.cc:2258] T 01c440b9aa6c49c587994c2c3298a31f P 9987807937db449a8d2cb8078493bc25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:17:39.032702 19559 ts_tablet_manager.cc:1905] T 01c440b9aa6c49c587994c2c3298a31f P 9987807937db449a8d2cb8078493bc25: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:17:39.034929 19559 ts_tablet_manager.cc:1918] T 01c440b9aa6c49c587994c2c3298a31f P 9987807937db449a8d2cb8078493bc25: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:17:39.035014 19559 log.cc:1196] T 01c440b9aa6c49c587994c2c3298a31f P 9987807937db449a8d2cb8078493bc25: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/ts-1/wal/wals/01c440b9aa6c49c587994c2c3298a31f
I20260924 04:17:39.035290 19559 ts_tablet_manager.cc:1939] T 01c440b9aa6c49c587994c2c3298a31f P 9987807937db449a8d2cb8078493bc25: Deleting consensus metadata
I20260924 04:17:39.036358 19560 tablet_replica.cc:330] T 01c440b9aa6c49c587994c2c3298a31f P 83ff1653f2ad4794a097905d3e60e21e: stopping tablet replica
I20260924 04:17:39.036330 19053 catalog_manager.cc:4841] TS 9987807937db449a8d2cb8078493bc25 (127.0.98.2:44855): tablet 01c440b9aa6c49c587994c2c3298a31f (table test-table [id=9faf8da19a454fac856881c3e3524dc9]) successfully deleted
I20260924 04:17:39.036649 19560 raft_consensus.cc:2229] T 01c440b9aa6c49c587994c2c3298a31f P 83ff1653f2ad4794a097905d3e60e21e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:17:39.036923 19560 raft_consensus.cc:2258] T 01c440b9aa6c49c587994c2c3298a31f P 83ff1653f2ad4794a097905d3e60e21e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:17:39.038288 19560 ts_tablet_manager.cc:1905] T 01c440b9aa6c49c587994c2c3298a31f P 83ff1653f2ad4794a097905d3e60e21e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 04:17:39.038533 19054 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44099 (error 108)
I20260924 04:17:39.039511 19531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:39.039244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44099 (local address 127.0.0.1:33140)
0924 04:17:39.039356 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:39.039426 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 04:17:39.039563 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 19258
I20260924 04:17:39.041170 19560 ts_tablet_manager.cc:1918] T 01c440b9aa6c49c587994c2c3298a31f P 83ff1653f2ad4794a097905d3e60e21e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:17:39.041256 19560 log.cc:1196] T 01c440b9aa6c49c587994c2c3298a31f P 83ff1653f2ad4794a097905d3e60e21e: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790223216265354-392-0/minicluster-data/ts-2/wal/wals/01c440b9aa6c49c587994c2c3298a31f
I20260924 04:17:39.041503 19560 ts_tablet_manager.cc:1939] T 01c440b9aa6c49c587994c2c3298a31f P 83ff1653f2ad4794a097905d3e60e21e: Deleting consensus metadata
W20260924 04:17:39.041934 19054 catalog_manager.cc:4584] TS db6a4816d61741e6a4affcc787215f39 (127.0.98.1:44099): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 01c440b9aa6c49c587994c2c3298a31f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44099: connect: Connection refused (error 111)
I20260924 04:17:39.042544 19053 catalog_manager.cc:4841] TS 83ff1653f2ad4794a097905d3e60e21e (127.0.98.3:45559): tablet 01c440b9aa6c49c587994c2c3298a31f (table test-table [id=9faf8da19a454fac856881c3e3524dc9]) successfully deleted
I20260924 04:17:39.045738 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 19389
I20260924 04:17:39.050207 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 19036
2026-09-24T04:17:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (949 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:17:39 dist-test-slave-p401 krb5kdc[19564](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:17:39 dist-test-slave-p401 krb5kdc[19564](info): set up 2 sockets
Sep 24 04:17:39 dist-test-slave-p401 krb5kdc[19564](info): commencing operation
krb5kdc: starting...
W20260924 04:17:39.986872 392 mini_kdc.cc:121] Time spent starting KDC: real 0.918s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/krb5kdc/test-user.keytab.
Sep 24 04:17:40 dist-test-slave-p401 krb5kdc[19564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:17:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:17:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:17:40.098465 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37987
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37987
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:17:40.204016 19577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:17:40.204273 19577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:17:40.204329 19577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:17:40.208256 19577 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:17:40.208308 19577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:17:40.208333 19577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:17:40.208353 19577 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:17:40.208382 19577 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:17:40.213533 19577 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37987
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:17:40.214524 19577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:17:40.215572 19577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:17:40.223644 19585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:17:40.223649 19583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:17:40.223730 19582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:17:40.223701 19577 server_base.cc:1034] running on GCE node
I20260924 04:17:40.224396 19577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:17:40.225400 19577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:17:40.226564 19577 hybrid_clock.cc:648] HybridClock initialized: now 1790223460226540 us; error 36 us; skew 500 ppm
Sep 24 04:17:40 dist-test-slave-p401 krb5kdc[19564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:17:40.229245 19577 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:17:40.230098 19577 webserver.cc:460] Webserver started at http://127.0.98.62:42269/ using document root <none> and password file <none>
I20260924 04:17:40.230616 19577 fs_manager.cc:359] Metadata directory not provided
I20260924 04:17:40.230682 19577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:17:40.230862 19577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:17:40.232846 19577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/master-0/data/instance:
uuid: "605d997cc6554bce89af74ab080962c3"
format_stamp: "Formatted at 2026-09-24 04:17:40 on dist-test-slave-p401"
server_key: "8d4a4df96e4e15170c6010538deb64c5"
server_key_iv: "b2e4cf145bfe1152cd4f75906da43c39"
server_key_version: "encryptionkey@0"
I20260924 04:17:40.233383 19577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/master-0/wal/instance:
uuid: "605d997cc6554bce89af74ab080962c3"
format_stamp: "Formatted at 2026-09-24 04:17:40 on dist-test-slave-p401"
server_key: "8d4a4df96e4e15170c6010538deb64c5"
server_key_iv: "b2e4cf145bfe1152cd4f75906da43c39"
server_key_version: "encryptionkey@0"
I20260924 04:17:40.237022 19577 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 04:17:40.239356 19592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:40.240522 19577 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:17:40.240630 19577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/master-0/wal
uuid: "605d997cc6554bce89af74ab080962c3"
format_stamp: "Formatted at 2026-09-24 04:17:40 on dist-test-slave-p401"
server_key: "8d4a4df96e4e15170c6010538deb64c5"
server_key_iv: "b2e4cf145bfe1152cd4f75906da43c39"
server_key_version: "encryptionkey@0"
I20260924 04:17:40.240723 19577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:17:40.254079 19577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:17:40.257057 19577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:17:40.257237 19577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:17:40.263952 19577 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37987
I20260924 04:17:40.263978 19644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37987 every 8 connection(s)
I20260924 04:17:40.264878 19577 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:17:40.267772 19645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:17:40.273332 19645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 605d997cc6554bce89af74ab080962c3: Bootstrap starting.
I20260924 04:17:40.273973 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 19577
I20260924 04:17:40.274082 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/master-0/wal/instance
I20260924 04:17:40.274312 392 external_mini_cluster.cc:1428] Setting key a76067d344643f3d264a3a79a7c14eef
I20260924 04:17:40.276083 19645 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 605d997cc6554bce89af74ab080962c3: Neither blocks nor log segments found. Creating new log.
I20260924 04:17:40.276804 19645 log.cc:824] T 00000000000000000000000000000000 P 605d997cc6554bce89af74ab080962c3: Log is configured to *not* fsync() on all Append() calls
I20260924 04:17:40.278618 19645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 605d997cc6554bce89af74ab080962c3: No bootstrap required, opened a new log
I20260924 04:17:40.282095 19645 raft_consensus.cc:348] T 00000000000000000000000000000000 P 605d997cc6554bce89af74ab080962c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "605d997cc6554bce89af74ab080962c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37987 } }
I20260924 04:17:40.282385 19645 raft_consensus.cc:374] T 00000000000000000000000000000000 P 605d997cc6554bce89af74ab080962c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
Sep 24 04:17:40 dist-test-slave-p401 krb5kdc[19564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:17:40.282447 19645 raft_consensus.cc:729] T 00000000000000000000000000000000 P 605d997cc6554bce89af74ab080962c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 605d997cc6554bce89af74ab080962c3, State: Initialized, Role: FOLLOWER
I20260924 04:17:40.282891 19645 consensus_queue.cc:260] T 00000000000000000000000000000000 P 605d997cc6554bce89af74ab080962c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "605d997cc6554bce89af74ab080962c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37987 } }
I20260924 04:17:40.283064 19645 raft_consensus.cc:388] T 00000000000000000000000000000000 P 605d997cc6554bce89af74ab080962c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:17:40.283142 19645 raft_consensus.cc:482] T 00000000000000000000000000000000 P 605d997cc6554bce89af74ab080962c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:17:40.283267 19645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 605d997cc6554bce89af74ab080962c3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:17:40.284391 19645 raft_consensus.cc:504] T 00000000000000000000000000000000 P 605d997cc6554bce89af74ab080962c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "605d997cc6554bce89af74ab080962c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37987 } }
I20260924 04:17:40.284818 19645 leader_election.cc:302] T 00000000000000000000000000000000 P 605d997cc6554bce89af74ab080962c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 605d997cc6554bce89af74ab080962c3; no voters:
I20260924 04:17:40.285177 19645 leader_election.cc:288] T 00000000000000000000000000000000 P 605d997cc6554bce89af74ab080962c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:17:40.285298 19650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 605d997cc6554bce89af74ab080962c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:17:40.285501 19650 raft_consensus.cc:686] T 00000000000000000000000000000000 P 605d997cc6554bce89af74ab080962c3 [term 1 LEADER]: Becoming Leader. State: Replica: 605d997cc6554bce89af74ab080962c3, State: Running, Role: LEADER
I20260924 04:17:40.285877 19650 consensus_queue.cc:237] T 00000000000000000000000000000000 P 605d997cc6554bce89af74ab080962c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "605d997cc6554bce89af74ab080962c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37987 } }
I20260924 04:17:40.286422 19645 sys_catalog.cc:564] T 00000000000000000000000000000000 P 605d997cc6554bce89af74ab080962c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:17:40.287528 19652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 605d997cc6554bce89af74ab080962c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "605d997cc6554bce89af74ab080962c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "605d997cc6554bce89af74ab080962c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37987 } } }
I20260924 04:17:40.287660 19652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 605d997cc6554bce89af74ab080962c3 [sys.catalog]: This master's current role is: LEADER
I20260924 04:17:40.287814 19651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 605d997cc6554bce89af74ab080962c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 605d997cc6554bce89af74ab080962c3. Latest consensus state: current_term: 1 leader_uuid: "605d997cc6554bce89af74ab080962c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "605d997cc6554bce89af74ab080962c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37987 } } }
I20260924 04:17:40.287966 19651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 605d997cc6554bce89af74ab080962c3 [sys.catalog]: This master's current role is: LEADER
I20260924 04:17:40.288256 19660 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:17:40.288501 19648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:40.275575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49402 (local address 127.0.98.62:37987)
0924 04:17:40.275997 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:40.276011 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:40.276045 (+ 34us) server_negotiation.cc:407] Connection header received
0924 04:17:40.276674 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:40.276694 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:40.277047 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:40.277354 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:17:40.278127 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:40.279115 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:40.279967 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:40.280427 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:40.280567 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:40.283006 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:17:40.283014 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:17:40.283031 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:17:40.283064 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:17:40.284980 (+ 1916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:17:40.285490 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:17:40.285497 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:17:40.285502 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:17:40.285645 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:17:40.285986 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:17:40.285989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:17:40.285991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:17:40.286478 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:17:40.286693 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:40.287117 (+ 424us) server_negotiation.cc:299] Negotiation successful
0924 04:17:40.287473 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":137,"threads_started":1}
I20260924 04:17:40.291213 19660 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:17:40.296450 19660 catalog_manager.cc:1347] Generated new cluster ID: 1c6b0c21ed8343c7bc1fd5f1d03d84a9
I20260924 04:17:40.296514 19660 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:17:40.305178 19660 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:17:40.306063 19660 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:17:40.313779 19660 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 605d997cc6554bce89af74ab080962c3: Generated new TSK 0
I20260924 04:17:40.314527 19660 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:17:40.342582 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37987
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:17:40.449770 19671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:17:40.450014 19671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:17:40.450070 19671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:17:40.454365 19671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:17:40.454445 19671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:17:40.454555 19671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:17:40.460037 19671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:17:40.461138 19671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:17:40.462153 19671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:17:40.471655 19677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:17:40.471657 19676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:17:40.471753 19671 server_base.cc:1034] running on GCE node
W20260924 04:17:40.471695 19679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:17:40.472369 19671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:17:40.473001 19671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:17:40.474154 19671 hybrid_clock.cc:648] HybridClock initialized: now 1790223460474131 us; error 37 us; skew 500 ppm
Sep 24 04:17:40 dist-test-slave-p401 krb5kdc[19564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:17:40.476824 19671 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:17:40.477754 19671 webserver.cc:460] Webserver started at http://127.0.98.1:46025/ using document root <none> and password file <none>
I20260924 04:17:40.478297 19671 fs_manager.cc:359] Metadata directory not provided
I20260924 04:17:40.478363 19671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:17:40.478559 19671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:17:40.480610 19671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/ts-0/data/instance:
uuid: "22033ce134ca48acb4494a579cedc26c"
format_stamp: "Formatted at 2026-09-24 04:17:40 on dist-test-slave-p401"
server_key: "72dbad68d5788518419f2918dcede7d4"
server_key_iv: "085d377dd90db6fe3b873581d4198343"
server_key_version: "encryptionkey@0"
I20260924 04:17:40.481142 19671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/ts-0/wal/instance:
uuid: "22033ce134ca48acb4494a579cedc26c"
format_stamp: "Formatted at 2026-09-24 04:17:40 on dist-test-slave-p401"
server_key: "72dbad68d5788518419f2918dcede7d4"
server_key_iv: "085d377dd90db6fe3b873581d4198343"
server_key_version: "encryptionkey@0"
I20260924 04:17:40.484813 19671 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:17:40.487315 19686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:40.488590 19671 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:17:40.488708 19671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/ts-0/wal
uuid: "22033ce134ca48acb4494a579cedc26c"
format_stamp: "Formatted at 2026-09-24 04:17:40 on dist-test-slave-p401"
server_key: "72dbad68d5788518419f2918dcede7d4"
server_key_iv: "085d377dd90db6fe3b873581d4198343"
server_key_version: "encryptionkey@0"
I20260924 04:17:40.488806 19671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:17:40.509563 19671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:17:40.512517 19671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:17:40.512681 19671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:17:40.513230 19671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:17:40.513989 19671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:17:40.514035 19671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:40.514086 19671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:17:40.514111 19671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:40.521986 19671 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44465
I20260924 04:17:40.522032 19799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44465 every 8 connection(s)
I20260924 04:17:40.522826 19671 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:17:40.528450 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 19671
I20260924 04:17:40.528580 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:17:40.528844 392 external_mini_cluster.cc:1428] Setting key 58f18742ff52af326bb50332f6c7cdfe
Sep 24 04:17:40 dist-test-slave-p401 krb5kdc[19564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:17:40.535800 19648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:40.524520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41091 (local address 127.0.98.62:37987)
0924 04:17:40.524634 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:40.524637 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:40.525456 (+ 819us) server_negotiation.cc:407] Connection header received
0924 04:17:40.526319 (+ 863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:40.526322 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:40.526379 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:40.526448 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:17:40.527616 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:40.528136 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:40.529358 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:40.529655 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:40.529742 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:40.532344 (+ 2602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:17:40.532348 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:17:40.532351 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:17:40.532380 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:17:40.533767 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:17:40.534262 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:17:40.534266 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:17:40.534268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:17:40.534357 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:17:40.534670 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:17:40.534673 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:17:40.534674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:17:40.534896 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:17:40.534996 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:40.535504 (+ 508us) server_negotiation.cc:299] Negotiation successful
0924 04:17:40.535631 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 04:17:40.536556 19802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:40.524889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37987 (local address 127.0.98.1:41091)
0924 04:17:40.525315 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:40.525355 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 04:17:40.526117 (+ 762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:17:40.526626 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:17:40.526633 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:17:40.526975 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:17:40.527481 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:40.527495 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:40.528263 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:40.528266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:40.529226 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:40.529238 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:40.529801 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:40.529804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:40.529984 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:40.530602 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:17:40.530619 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:17:40.532181 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:17:40.533869 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:17:40.533873 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:17:40.533886 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:17:40.534160 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:17:40.534441 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:17:40.534443 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:17:40.534444 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:17:40.534590 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:17:40.534989 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:17:40.534994 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:17:40.535271 (+ 277us) client_negotiation.cc:769] Sending connection context
0924 04:17:40.535438 (+ 167us) client_negotiation.cc:240] Negotiation successful
0924 04:17:40.535711 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":109,"threads_started":1}
I20260924 04:17:40.537703 19800 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37987
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:17:40.537925 19800 heartbeater.cc:464] Registering TS with master...
I20260924 04:17:40.538414 19800 heartbeater.cc:511] Master 127.0.98.62:37987 requested a full tablet report, sending...
I20260924 04:17:40.539852 19609 ts_manager.cc:194] Registered new tserver with Master: 22033ce134ca48acb4494a579cedc26c (127.0.98.1:44465)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:17:40.541237 19609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41091
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:17:40.554512 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37987
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:17:40.660413 19805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:17:40.660666 19805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:17:40.660715 19805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:17:40.664666 19805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:17:40.664726 19805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:17:40.664784 19805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:17:40.670128 19805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:17:40.671118 19805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:17:40.672168 19805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:17:40.681291 19810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:17:40.681387 19805 server_base.cc:1034] running on GCE node
W20260924 04:17:40.681341 19811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:17:40.681301 19813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:17:40.681953 19805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:17:40.682595 19805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:17:40.683744 19805 hybrid_clock.cc:648] HybridClock initialized: now 1790223460683710 us; error 45 us; skew 500 ppm
Sep 24 04:17:40 dist-test-slave-p401 krb5kdc[19564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:17:40.686305 19805 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:17:40.687151 19805 webserver.cc:460] Webserver started at http://127.0.98.2:36491/ using document root <none> and password file <none>
I20260924 04:17:40.687754 19805 fs_manager.cc:359] Metadata directory not provided
I20260924 04:17:40.687821 19805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:17:40.688021 19805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:17:40.689958 19805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/ts-1/data/instance:
uuid: "7675d8f361574350ad626141be4594d9"
format_stamp: "Formatted at 2026-09-24 04:17:40 on dist-test-slave-p401"
server_key: "396f650131c9da68c6cf4bfc3d98bc39"
server_key_iv: "bc611424854cf43b63af3c509c941fc5"
server_key_version: "encryptionkey@0"
I20260924 04:17:40.690469 19805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7675d8f361574350ad626141be4594d9"
format_stamp: "Formatted at 2026-09-24 04:17:40 on dist-test-slave-p401"
server_key: "396f650131c9da68c6cf4bfc3d98bc39"
server_key_iv: "bc611424854cf43b63af3c509c941fc5"
server_key_version: "encryptionkey@0"
I20260924 04:17:40.694038 19805 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260924 04:17:40.696281 19820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:40.697363 19805 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:17:40.697465 19805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/ts-1/wal
uuid: "7675d8f361574350ad626141be4594d9"
format_stamp: "Formatted at 2026-09-24 04:17:40 on dist-test-slave-p401"
server_key: "396f650131c9da68c6cf4bfc3d98bc39"
server_key_iv: "bc611424854cf43b63af3c509c941fc5"
server_key_version: "encryptionkey@0"
I20260924 04:17:40.697551 19805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:17:40.716266 19805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:17:40.719074 19805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:17:40.719269 19805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:17:40.719852 19805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:17:40.720654 19805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:17:40.720707 19805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:40.720758 19805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:17:40.720788 19805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:40.730830 19805 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42143
I20260924 04:17:40.730858 19933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42143 every 8 connection(s)
I20260924 04:17:40.731813 19805 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:17:40.740401 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 19805
I20260924 04:17:40.740545 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/ts-1/wal/instance
Sep 24 04:17:40 dist-test-slave-p401 krb5kdc[19564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:17:40.740862 392 external_mini_cluster.cc:1428] Setting key 13454f2b1be3f042ece561d617b29613
I20260924 04:17:40.744539 19648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:40.733545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60291 (local address 127.0.98.62:37987)
0924 04:17:40.733669 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:40.733672 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:40.734540 (+ 868us) server_negotiation.cc:407] Connection header received
0924 04:17:40.735456 (+ 916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:40.735459 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:40.735513 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:40.735584 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:17:40.736723 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:40.737231 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:40.738321 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:40.738533 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:40.738617 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:40.741176 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:17:40.741180 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:17:40.741182 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:17:40.741210 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:17:40.742673 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:17:40.743121 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:17:40.743125 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:17:40.743126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:17:40.743201 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:17:40.743516 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:17:40.743519 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:17:40.743521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:17:40.743698 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:17:40.743784 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:40.744281 (+ 497us) server_negotiation.cc:299] Negotiation successful
0924 04:17:40.744387 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 04:17:40.745460 19936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:40.733929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37987 (local address 127.0.98.2:60291)
0924 04:17:40.734397 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:40.734438 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 04:17:40.735223 (+ 785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:17:40.735767 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:17:40.735773 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:17:40.736111 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:17:40.736579 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:40.736592 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:40.737341 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:40.737345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:40.738211 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:40.738220 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:40.738663 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:40.738667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:40.738838 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:40.739416 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:17:40.739436 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:17:40.741030 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:17:40.742779 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:17:40.742785 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:17:40.742800 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:17:40.743035 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:17:40.743295 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:17:40.743298 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:17:40.743301 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:17:40.743435 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:17:40.743785 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:17:40.743791 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:17:40.744060 (+ 269us) client_negotiation.cc:769] Sending connection context
0924 04:17:40.744231 (+ 171us) client_negotiation.cc:240] Negotiation successful
0924 04:17:40.744539 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":148,"threads_started":1}
I20260924 04:17:40.746523 19934 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37987
I20260924 04:17:40.746793 19934 heartbeater.cc:464] Registering TS with master...
I20260924 04:17:40.747498 19934 heartbeater.cc:511] Master 127.0.98.62:37987 requested a full tablet report, sending...
I20260924 04:17:40.748744 19609 ts_manager.cc:194] Registered new tserver with Master: 7675d8f361574350ad626141be4594d9 (127.0.98.2:42143)
I20260924 04:17:40.749296 19609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60291
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:17:40.768312 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37987
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:17:40.876466 19939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:17:40.876730 19939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:17:40.876771 19939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:17:40.880684 19939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:17:40.880734 19939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:17:40.880790 19939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:17:40.885871 19939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:17:40.886827 19939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:17:40.887871 19939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:17:40.897125 19945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:17:40.897125 19944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:17:40.897185 19939 server_base.cc:1034] running on GCE node
W20260924 04:17:40.897131 19947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:17:40.897802 19939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:17:40.898415 19939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:17:40.899592 19939 hybrid_clock.cc:648] HybridClock initialized: now 1790223460899566 us; error 40 us; skew 500 ppm
Sep 24 04:17:40 dist-test-slave-p401 krb5kdc[19564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:17:40.902486 19939 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:17:40.903388 19939 webserver.cc:460] Webserver started at http://127.0.98.3:46659/ using document root <none> and password file <none>
I20260924 04:17:40.903944 19939 fs_manager.cc:359] Metadata directory not provided
I20260924 04:17:40.904009 19939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:17:40.904220 19939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:17:40.906221 19939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/ts-2/data/instance:
uuid: "ad83e16c91c34404a8e16cc4705b069a"
format_stamp: "Formatted at 2026-09-24 04:17:40 on dist-test-slave-p401"
server_key: "2afd75b75205a872af8e12f6ea231aab"
server_key_iv: "543b138bd70e5ae0b020b80b960dc20f"
server_key_version: "encryptionkey@0"
I20260924 04:17:40.906749 19939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ad83e16c91c34404a8e16cc4705b069a"
format_stamp: "Formatted at 2026-09-24 04:17:40 on dist-test-slave-p401"
server_key: "2afd75b75205a872af8e12f6ea231aab"
server_key_iv: "543b138bd70e5ae0b020b80b960dc20f"
server_key_version: "encryptionkey@0"
I20260924 04:17:40.910243 19939 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 04:17:40.912664 19954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:40.913931 19939 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:17:40.914083 19939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/ts-2/wal
uuid: "ad83e16c91c34404a8e16cc4705b069a"
format_stamp: "Formatted at 2026-09-24 04:17:40 on dist-test-slave-p401"
server_key: "2afd75b75205a872af8e12f6ea231aab"
server_key_iv: "543b138bd70e5ae0b020b80b960dc20f"
server_key_version: "encryptionkey@0"
I20260924 04:17:40.914183 19939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:17:40.939137 19939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:17:40.942286 19939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:17:40.942461 19939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:17:40.943025 19939 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:17:40.943900 19939 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:17:40.943949 19939 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:40.944002 19939 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:17:40.944027 19939 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:40.952373 19939 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46183
I20260924 04:17:40.952407 20067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46183 every 8 connection(s)
I20260924 04:17:40.953261 19939 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:17:40.954399 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 19939
I20260924 04:17:40.954490 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:17:40.954752 392 external_mini_cluster.cc:1428] Setting key 00d75f9d782f825885a438dcc0093081
Sep 24 04:17:40 dist-test-slave-p401 krb5kdc[19564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:17:40.966202 19648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:40.955409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58017 (local address 127.0.98.62:37987)
0924 04:17:40.955535 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:40.955539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:40.956124 (+ 585us) server_negotiation.cc:407] Connection header received
0924 04:17:40.957454 (+ 1330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:40.957457 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:40.957508 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:40.957585 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:17:40.958701 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:40.959220 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:40.960335 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:40.960550 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:40.960641 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:40.962964 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:17:40.962967 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:17:40.962969 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:17:40.962996 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:17:40.964431 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:17:40.964887 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:17:40.964890 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:17:40.964891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:17:40.964964 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:17:40.965228 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:17:40.965230 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:17:40.965232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:17:40.965388 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:17:40.965456 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:40.965975 (+ 519us) server_negotiation.cc:299] Negotiation successful
0924 04:17:40.966081 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:17:40.966995 20070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:40.955477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37987 (local address 127.0.98.3:58017)
0924 04:17:40.955984 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:40.956028 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 04:17:40.957269 (+ 1241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:17:40.957745 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:17:40.957752 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:17:40.958099 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:17:40.958576 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:40.958589 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:40.959349 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:40.959352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:40.960221 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:40.960230 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:40.960687 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:40.960691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:40.960848 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:40.961358 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:17:40.961376 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:17:40.962813 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:17:40.964546 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:17:40.964551 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:17:40.964562 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:17:40.964807 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:17:40.965029 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:17:40.965032 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:17:40.965033 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:17:40.965165 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:17:40.965491 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:17:40.965496 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:17:40.965731 (+ 235us) client_negotiation.cc:769] Sending connection context
0924 04:17:40.965895 (+ 164us) client_negotiation.cc:240] Negotiation successful
0924 04:17:40.966185 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":140,"threads_started":1}
I20260924 04:17:40.968205 20068 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37987
I20260924 04:17:40.968430 20068 heartbeater.cc:464] Registering TS with master...
I20260924 04:17:40.968888 20068 heartbeater.cc:511] Master 127.0.98.62:37987 requested a full tablet report, sending...
I20260924 04:17:40.969815 19608 ts_manager.cc:194] Registered new tserver with Master: ad83e16c91c34404a8e16cc4705b069a (127.0.98.3:46183)
I20260924 04:17:40.970412 19608 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58017
I20260924 04:17:40.980621 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:17:40.989599 19648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:40.982423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49418 (local address 127.0.98.62:37987)
0924 04:17:40.982574 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:40.982578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:40.982705 (+ 127us) server_negotiation.cc:407] Connection header received
0924 04:17:40.982827 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:40.982830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:40.982875 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:40.982938 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:17:40.983666 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:40.984196 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:40.985000 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:40.985217 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:40.985301 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:40.986622 (+ 1321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:17:40.986626 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:17:40.986629 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:17:40.986658 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:17:40.988080 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:17:40.988445 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:17:40.988449 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:17:40.988451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:17:40.988529 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:17:40.988784 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:17:40.988787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:17:40.988789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:17:40.989004 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:17:40.989134 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:40.989344 (+ 210us) server_negotiation.cc:299] Negotiation successful
0924 04:17:40.989449 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 04:17:40.992450 19608 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49418:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:17:40.995455 19608 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:17:41.009168 20079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:41.003468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42143 (local address 127.0.0.1:39626)
0924 04:17:41.004242 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:41.004291 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 04:17:41.004490 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:17:41.005205 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:17:41.005210 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:17:41.005229 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:17:41.005340 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:41.005346 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:41.006447 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:41.006450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:41.007714 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:41.007730 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:41.008619 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:41.008623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:41.008735 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:41.008770 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 04:17:41.008876 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 04:17:41.009004 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"spinlock_wait_cycles":6144,"thread_start_us":71,"threads_started":1}
I20260924 04:17:41.009173 20081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:41.003576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39626 (local address 127.0.98.2:42143)
0924 04:17:41.004210 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:41.004224 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:41.004373 (+ 149us) server_negotiation.cc:407] Connection header received
0924 04:17:41.004680 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:41.004686 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:41.004874 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:41.005020 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:17:41.005458 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:41.006326 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:41.007869 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:41.008506 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:41.008610 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:41.008769 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:41.008918 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 04:17:41.009011 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":106,"threads_started":1}
I20260924 04:17:41.009760 20083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:41.004231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46183 (local address 127.0.0.1:55414)
0924 04:17:41.005228 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:41.005239 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:17:41.005312 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:17:41.005678 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:17:41.005680 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:17:41.005698 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:17:41.005789 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:41.005794 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:41.007043 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:41.007046 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:41.008070 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:41.008080 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:41.009256 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:41.009260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:41.009450 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:41.009505 (+ 55us) client_negotiation.cc:769] Sending connection context
0924 04:17:41.009566 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 04:17:41.009632 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":946,"mutex_wait_us":76,"thread_start_us":93,"threads_started":1}
I20260924 04:17:41.010043 20080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:41.003512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44465 (local address 127.0.0.1:39994)
0924 04:17:41.004343 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:41.004355 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:17:41.004495 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:17:41.005374 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:17:41.005376 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:17:41.005384 (+ 8us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:17:41.005471 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:41.005476 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:41.006859 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:41.006862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:41.008020 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:41.008028 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:41.009256 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:41.009260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:41.009434 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:41.009500 (+ 66us) client_negotiation.cc:769] Sending connection context
0924 04:17:41.009842 (+ 342us) client_negotiation.cc:240] Negotiation successful
0924 04:17:41.009915 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"mutex_wait_us":61,"spinlock_wait_cycles":10240,"thread_start_us":86,"threads_started":1}
I20260924 04:17:41.010160 20082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:41.003733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39994 (local address 127.0.98.1:44465)
0924 04:17:41.004562 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:41.004602 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:41.004622 (+ 20us) server_negotiation.cc:407] Connection header received
0924 04:17:41.004688 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:41.004692 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:41.004849 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:41.004987 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:17:41.005580 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:41.006727 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:41.008131 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:41.008968 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:41.009068 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:41.009195 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:41.009896 (+ 701us) server_negotiation.cc:299] Negotiation successful
0924 04:17:41.010019 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":731,"thread_start_us":90,"threads_started":1}
I20260924 04:17:41.012457 19734 tablet_service.cc:1467] Processing CreateTablet for tablet c228b5f32a6f4709a1affb9e17d1b892 (DEFAULT_TABLE table=test-table [id=584837b57bf94df585db0a10047ee8e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:17:41.012444 19868 tablet_service.cc:1467] Processing CreateTablet for tablet c228b5f32a6f4709a1affb9e17d1b892 (DEFAULT_TABLE table=test-table [id=584837b57bf94df585db0a10047ee8e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:17:41.013228 20084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:41.004473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55414 (local address 127.0.98.3:46183)
0924 04:17:41.005073 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:41.005084 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:41.005317 (+ 233us) server_negotiation.cc:407] Connection header received
0924 04:17:41.005391 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:41.005396 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:41.005555 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:41.005702 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:17:41.005873 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:41.006937 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:41.008206 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:41.008968 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:41.009068 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:41.009203 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:41.012962 (+ 3759us) server_negotiation.cc:299] Negotiation successful
0924 04:17:41.013077 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":93,"threads_started":1}
I20260924 04:17:41.013433 19734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c228b5f32a6f4709a1affb9e17d1b892. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:17:41.013433 19868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c228b5f32a6f4709a1affb9e17d1b892. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:17:41.016135 20002 tablet_service.cc:1467] Processing CreateTablet for tablet c228b5f32a6f4709a1affb9e17d1b892 (DEFAULT_TABLE table=test-table [id=584837b57bf94df585db0a10047ee8e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:17:41.017179 20002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c228b5f32a6f4709a1affb9e17d1b892. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:17:41.020196 20085 tablet_bootstrap.cc:492] T c228b5f32a6f4709a1affb9e17d1b892 P 7675d8f361574350ad626141be4594d9: Bootstrap starting.
I20260924 04:17:41.021008 20086 tablet_bootstrap.cc:492] T c228b5f32a6f4709a1affb9e17d1b892 P 22033ce134ca48acb4494a579cedc26c: Bootstrap starting.
I20260924 04:17:41.022384 20085 tablet_bootstrap.cc:654] T c228b5f32a6f4709a1affb9e17d1b892 P 7675d8f361574350ad626141be4594d9: Neither blocks nor log segments found. Creating new log.
I20260924 04:17:41.023096 20086 tablet_bootstrap.cc:654] T c228b5f32a6f4709a1affb9e17d1b892 P 22033ce134ca48acb4494a579cedc26c: Neither blocks nor log segments found. Creating new log.
I20260924 04:17:41.023401 20085 log.cc:824] T c228b5f32a6f4709a1affb9e17d1b892 P 7675d8f361574350ad626141be4594d9: Log is configured to *not* fsync() on all Append() calls
I20260924 04:17:41.023926 20087 tablet_bootstrap.cc:492] T c228b5f32a6f4709a1affb9e17d1b892 P ad83e16c91c34404a8e16cc4705b069a: Bootstrap starting.
I20260924 04:17:41.024248 20086 log.cc:824] T c228b5f32a6f4709a1affb9e17d1b892 P 22033ce134ca48acb4494a579cedc26c: Log is configured to *not* fsync() on all Append() calls
I20260924 04:17:41.025483 20085 tablet_bootstrap.cc:492] T c228b5f32a6f4709a1affb9e17d1b892 P 7675d8f361574350ad626141be4594d9: No bootstrap required, opened a new log
I20260924 04:17:41.025663 20085 ts_tablet_manager.cc:1397] T c228b5f32a6f4709a1affb9e17d1b892 P 7675d8f361574350ad626141be4594d9: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:17:41.026099 20086 tablet_bootstrap.cc:492] T c228b5f32a6f4709a1affb9e17d1b892 P 22033ce134ca48acb4494a579cedc26c: No bootstrap required, opened a new log
I20260924 04:17:41.026228 20086 ts_tablet_manager.cc:1397] T c228b5f32a6f4709a1affb9e17d1b892 P 22033ce134ca48acb4494a579cedc26c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 04:17:41.026420 20087 tablet_bootstrap.cc:654] T c228b5f32a6f4709a1affb9e17d1b892 P ad83e16c91c34404a8e16cc4705b069a: Neither blocks nor log segments found. Creating new log.
I20260924 04:17:41.027597 20087 log.cc:824] T c228b5f32a6f4709a1affb9e17d1b892 P ad83e16c91c34404a8e16cc4705b069a: Log is configured to *not* fsync() on all Append() calls
I20260924 04:17:41.029132 20086 raft_consensus.cc:348] T c228b5f32a6f4709a1affb9e17d1b892 P 22033ce134ca48acb4494a579cedc26c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7675d8f361574350ad626141be4594d9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42143 } } peers { permanent_uuid: "22033ce134ca48acb4494a579cedc26c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44465 } } peers { permanent_uuid: "ad83e16c91c34404a8e16cc4705b069a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46183 } }
I20260924 04:17:41.029346 20086 raft_consensus.cc:374] T c228b5f32a6f4709a1affb9e17d1b892 P 22033ce134ca48acb4494a579cedc26c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:17:41.029419 20086 raft_consensus.cc:729] T c228b5f32a6f4709a1affb9e17d1b892 P 22033ce134ca48acb4494a579cedc26c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22033ce134ca48acb4494a579cedc26c, State: Initialized, Role: FOLLOWER
I20260924 04:17:41.029495 20087 tablet_bootstrap.cc:492] T c228b5f32a6f4709a1affb9e17d1b892 P ad83e16c91c34404a8e16cc4705b069a: No bootstrap required, opened a new log
I20260924 04:17:41.029403 20085 raft_consensus.cc:348] T c228b5f32a6f4709a1affb9e17d1b892 P 7675d8f361574350ad626141be4594d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7675d8f361574350ad626141be4594d9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42143 } } peers { permanent_uuid: "22033ce134ca48acb4494a579cedc26c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44465 } } peers { permanent_uuid: "ad83e16c91c34404a8e16cc4705b069a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46183 } }
I20260924 04:17:41.029670 20085 raft_consensus.cc:374] T c228b5f32a6f4709a1affb9e17d1b892 P 7675d8f361574350ad626141be4594d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:17:41.029690 20087 ts_tablet_manager.cc:1397] T c228b5f32a6f4709a1affb9e17d1b892 P ad83e16c91c34404a8e16cc4705b069a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:17:41.029719 20085 raft_consensus.cc:729] T c228b5f32a6f4709a1affb9e17d1b892 P 7675d8f361574350ad626141be4594d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7675d8f361574350ad626141be4594d9, State: Initialized, Role: FOLLOWER
I20260924 04:17:41.029878 20086 consensus_queue.cc:260] T c228b5f32a6f4709a1affb9e17d1b892 P 22033ce134ca48acb4494a579cedc26c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7675d8f361574350ad626141be4594d9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42143 } } peers { permanent_uuid: "22033ce134ca48acb4494a579cedc26c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44465 } } peers { permanent_uuid: "ad83e16c91c34404a8e16cc4705b069a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46183 } }
I20260924 04:17:41.030157 20085 consensus_queue.cc:260] T c228b5f32a6f4709a1affb9e17d1b892 P 7675d8f361574350ad626141be4594d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7675d8f361574350ad626141be4594d9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42143 } } peers { permanent_uuid: "22033ce134ca48acb4494a579cedc26c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44465 } } peers { permanent_uuid: "ad83e16c91c34404a8e16cc4705b069a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46183 } }
I20260924 04:17:41.030475 19800 heartbeater.cc:503] Master 127.0.98.62:37987 was elected leader, sending a full tablet report...
I20260924 04:17:41.030804 20086 ts_tablet_manager.cc:1428] T c228b5f32a6f4709a1affb9e17d1b892 P 22033ce134ca48acb4494a579cedc26c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 04:17:41.031208 20085 ts_tablet_manager.cc:1428] T c228b5f32a6f4709a1affb9e17d1b892 P 7675d8f361574350ad626141be4594d9: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 04:17:41.031576 19934 heartbeater.cc:503] Master 127.0.98.62:37987 was elected leader, sending a full tablet report...
I20260924 04:17:41.033429 20087 raft_consensus.cc:348] T c228b5f32a6f4709a1affb9e17d1b892 P ad83e16c91c34404a8e16cc4705b069a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7675d8f361574350ad626141be4594d9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42143 } } peers { permanent_uuid: "22033ce134ca48acb4494a579cedc26c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44465 } } peers { permanent_uuid: "ad83e16c91c34404a8e16cc4705b069a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46183 } }
I20260924 04:17:41.033720 20087 raft_consensus.cc:374] T c228b5f32a6f4709a1affb9e17d1b892 P ad83e16c91c34404a8e16cc4705b069a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:17:41.033782 20087 raft_consensus.cc:729] T c228b5f32a6f4709a1affb9e17d1b892 P ad83e16c91c34404a8e16cc4705b069a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad83e16c91c34404a8e16cc4705b069a, State: Initialized, Role: FOLLOWER
I20260924 04:17:41.034186 20087 consensus_queue.cc:260] T c228b5f32a6f4709a1affb9e17d1b892 P ad83e16c91c34404a8e16cc4705b069a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7675d8f361574350ad626141be4594d9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42143 } } peers { permanent_uuid: "22033ce134ca48acb4494a579cedc26c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44465 } } peers { permanent_uuid: "ad83e16c91c34404a8e16cc4705b069a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46183 } }
I20260924 04:17:41.034751 20068 heartbeater.cc:503] Master 127.0.98.62:37987 was elected leader, sending a full tablet report...
I20260924 04:17:41.035143 20087 ts_tablet_manager.cc:1428] T c228b5f32a6f4709a1affb9e17d1b892 P ad83e16c91c34404a8e16cc4705b069a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260924 04:17:41.204157 20069 tablet.cc:2369] T c228b5f32a6f4709a1affb9e17d1b892 P ad83e16c91c34404a8e16cc4705b069a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:17:41.232892 19935 tablet.cc:2369] T c228b5f32a6f4709a1affb9e17d1b892 P 7675d8f361574350ad626141be4594d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:17:41.274092 19801 tablet.cc:2369] T c228b5f32a6f4709a1affb9e17d1b892 P 22033ce134ca48acb4494a579cedc26c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:17:41.291867 20092 raft_consensus.cc:482] T c228b5f32a6f4709a1affb9e17d1b892 P 7675d8f361574350ad626141be4594d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:17:41.292091 20092 raft_consensus.cc:504] T c228b5f32a6f4709a1affb9e17d1b892 P 7675d8f361574350ad626141be4594d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7675d8f361574350ad626141be4594d9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42143 } } peers { permanent_uuid: "22033ce134ca48acb4494a579cedc26c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44465 } } peers { permanent_uuid: "ad83e16c91c34404a8e16cc4705b069a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46183 } }
I20260924 04:17:41.293387 20092 leader_election.cc:288] T c228b5f32a6f4709a1affb9e17d1b892 P 7675d8f361574350ad626141be4594d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22033ce134ca48acb4494a579cedc26c (127.0.98.1:44465), ad83e16c91c34404a8e16cc4705b069a (127.0.98.3:46183)
I20260924 04:17:41.297060 20082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:41.293658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35973 (local address 127.0.98.1:44465)
0924 04:17:41.293797 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:41.293801 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:41.294035 (+ 234us) server_negotiation.cc:407] Connection header received
0924 04:17:41.294151 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:41.294155 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:41.294204 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:41.294261 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:17:41.294527 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:41.295033 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:41.296185 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:41.296554 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:41.296631 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:41.296704 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:41.296865 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 04:17:41.296940 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 04:17:41.297082 20095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:41.293557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44465 (local address 127.0.98.2:35973)
0924 04:17:41.293932 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:41.293946 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:17:41.294049 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:17:41.294285 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:17:41.294288 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:17:41.294304 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:17:41.294428 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:41.294438 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:41.295137 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:41.295140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:41.296072 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:41.296081 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:41.296661 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:41.296665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:41.296755 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:41.296778 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 04:17:41.296843 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 04:17:41.296903 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":19,"thread_start_us":65,"threads_started":1}
I20260924 04:17:41.297102 20084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:41.293756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44009 (local address 127.0.98.3:46183)
0924 04:17:41.293939 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:41.293944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:41.294086 (+ 142us) server_negotiation.cc:407] Connection header received
0924 04:17:41.294199 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:41.294202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:41.294246 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:41.294316 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:17:41.294563 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:41.295031 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:41.296184 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:41.296595 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:41.296694 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:41.296752 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:41.296923 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 04:17:41.296991 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260924 04:17:41.297093 20094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:41.293645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46183 (local address 127.0.98.2:44009)
0924 04:17:41.293993 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:41.294006 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:17:41.294106 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:17:41.294334 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:17:41.294337 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:17:41.294352 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:17:41.294456 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:41.294461 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:41.295164 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:41.295167 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:41.296064 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:41.296073 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:41.296704 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:41.296708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:41.296807 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:41.296826 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 04:17:41.296896 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 04:17:41.296955 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":27,"thread_start_us":91,"threads_started":1}
I20260924 04:17:41.297770 20022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c228b5f32a6f4709a1affb9e17d1b892" candidate_uuid: "7675d8f361574350ad626141be4594d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad83e16c91c34404a8e16cc4705b069a" is_pre_election: true
I20260924 04:17:41.297897 19754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c228b5f32a6f4709a1affb9e17d1b892" candidate_uuid: "7675d8f361574350ad626141be4594d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22033ce134ca48acb4494a579cedc26c" is_pre_election: true
I20260924 04:17:41.298095 20022 raft_consensus.cc:2454] T c228b5f32a6f4709a1affb9e17d1b892 P ad83e16c91c34404a8e16cc4705b069a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7675d8f361574350ad626141be4594d9 in term 0.
I20260924 04:17:41.298126 19754 raft_consensus.cc:2454] T c228b5f32a6f4709a1affb9e17d1b892 P 22033ce134ca48acb4494a579cedc26c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7675d8f361574350ad626141be4594d9 in term 0.
I20260924 04:17:41.298595 19821 leader_election.cc:302] T c228b5f32a6f4709a1affb9e17d1b892 P 7675d8f361574350ad626141be4594d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7675d8f361574350ad626141be4594d9, ad83e16c91c34404a8e16cc4705b069a; no voters:
I20260924 04:17:41.298842 20092 raft_consensus.cc:2781] T c228b5f32a6f4709a1affb9e17d1b892 P 7675d8f361574350ad626141be4594d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:17:41.298918 20092 raft_consensus.cc:482] T c228b5f32a6f4709a1affb9e17d1b892 P 7675d8f361574350ad626141be4594d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:17:41.298969 20092 raft_consensus.cc:3037] T c228b5f32a6f4709a1affb9e17d1b892 P 7675d8f361574350ad626141be4594d9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:17:41.299989 20092 raft_consensus.cc:504] T c228b5f32a6f4709a1affb9e17d1b892 P 7675d8f361574350ad626141be4594d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7675d8f361574350ad626141be4594d9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42143 } } peers { permanent_uuid: "22033ce134ca48acb4494a579cedc26c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44465 } } peers { permanent_uuid: "ad83e16c91c34404a8e16cc4705b069a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46183 } }
I20260924 04:17:41.300292 20092 leader_election.cc:288] T c228b5f32a6f4709a1affb9e17d1b892 P 7675d8f361574350ad626141be4594d9 [CANDIDATE]: Term 1 election: Requested vote from peers 22033ce134ca48acb4494a579cedc26c (127.0.98.1:44465), ad83e16c91c34404a8e16cc4705b069a (127.0.98.3:46183)
I20260924 04:17:41.300606 19754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c228b5f32a6f4709a1affb9e17d1b892" candidate_uuid: "7675d8f361574350ad626141be4594d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22033ce134ca48acb4494a579cedc26c"
I20260924 04:17:41.300717 19754 raft_consensus.cc:3037] T c228b5f32a6f4709a1affb9e17d1b892 P 22033ce134ca48acb4494a579cedc26c [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:17:41.301790 19754 raft_consensus.cc:2454] T c228b5f32a6f4709a1affb9e17d1b892 P 22033ce134ca48acb4494a579cedc26c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7675d8f361574350ad626141be4594d9 in term 1.
I20260924 04:17:41.302024 20022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c228b5f32a6f4709a1affb9e17d1b892" candidate_uuid: "7675d8f361574350ad626141be4594d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad83e16c91c34404a8e16cc4705b069a"
I20260924 04:17:41.302080 19824 leader_election.cc:302] T c228b5f32a6f4709a1affb9e17d1b892 P 7675d8f361574350ad626141be4594d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22033ce134ca48acb4494a579cedc26c, 7675d8f361574350ad626141be4594d9; no voters:
I20260924 04:17:41.302155 20022 raft_consensus.cc:3037] T c228b5f32a6f4709a1affb9e17d1b892 P ad83e16c91c34404a8e16cc4705b069a [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:17:41.302235 20092 raft_consensus.cc:2781] T c228b5f32a6f4709a1affb9e17d1b892 P 7675d8f361574350ad626141be4594d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:17:41.302429 20092 raft_consensus.cc:686] T c228b5f32a6f4709a1affb9e17d1b892 P 7675d8f361574350ad626141be4594d9 [term 1 LEADER]: Becoming Leader. State: Replica: 7675d8f361574350ad626141be4594d9, State: Running, Role: LEADER
I20260924 04:17:41.302757 20092 consensus_queue.cc:237] T c228b5f32a6f4709a1affb9e17d1b892 P 7675d8f361574350ad626141be4594d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7675d8f361574350ad626141be4594d9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42143 } } peers { permanent_uuid: "22033ce134ca48acb4494a579cedc26c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44465 } } peers { permanent_uuid: "ad83e16c91c34404a8e16cc4705b069a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46183 } }
I20260924 04:17:41.303177 20022 raft_consensus.cc:2454] T c228b5f32a6f4709a1affb9e17d1b892 P ad83e16c91c34404a8e16cc4705b069a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7675d8f361574350ad626141be4594d9 in term 1.
I20260924 04:17:41.305744 19608 catalog_manager.cc:5495] T c228b5f32a6f4709a1affb9e17d1b892 P 7675d8f361574350ad626141be4594d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7675d8f361574350ad626141be4594d9 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "7675d8f361574350ad626141be4594d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7675d8f361574350ad626141be4594d9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22033ce134ca48acb4494a579cedc26c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad83e16c91c34404a8e16cc4705b069a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46183 } health_report { overall_health: UNKNOWN } } }
I20260924 04:17:41.370932 20081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:41.367669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39630 (local address 127.0.98.2:42143)
0924 04:17:41.367799 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:41.367803 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:41.367819 (+ 16us) server_negotiation.cc:407] Connection header received
0924 04:17:41.367915 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:41.367917 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:41.367980 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:41.368084 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:17:41.368302 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:41.368847 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:41.369651 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:41.369908 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:41.369999 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:41.370212 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:17:41.370635 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:17:41.370730 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:41.370769 (+ 39us) server_negotiation.cc:299] Negotiation successful
0924 04:17:41.370816 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:17:41.376495 20022 raft_consensus.cc:1261] T c228b5f32a6f4709a1affb9e17d1b892 P ad83e16c91c34404a8e16cc4705b069a [term 1 FOLLOWER]: Refusing update from remote peer 7675d8f361574350ad626141be4594d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:17:41.376495 19754 raft_consensus.cc:1261] T c228b5f32a6f4709a1affb9e17d1b892 P 22033ce134ca48acb4494a579cedc26c [term 1 FOLLOWER]: Refusing update from remote peer 7675d8f361574350ad626141be4594d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:17:41.377169 20096 consensus_queue.cc:1035] T c228b5f32a6f4709a1affb9e17d1b892 P 7675d8f361574350ad626141be4594d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad83e16c91c34404a8e16cc4705b069a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:17:41.377341 20092 consensus_queue.cc:1035] T c228b5f32a6f4709a1affb9e17d1b892 P 7675d8f361574350ad626141be4594d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22033ce134ca48acb4494a579cedc26c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:17:41.384533 20101 mvcc.cc:205] Tried to move back new op lower bound from 7332755297791471616 to 7332755297498902528. Current Snapshot: MvccSnapshot[applied={T|T < 7332755297791471616}]
I20260924 04:17:41.384936 20100 mvcc.cc:205] Tried to move back new op lower bound from 7332755297791471616 to 7332755297498902528. Current Snapshot: MvccSnapshot[applied={T|T < 7332755297791471616}]
I20260924 04:17:41.392525 19608 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:17:41.392683 19608 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:17:41.395008 19608 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 605d997cc6554bce89af74ab080962c3: Sending DeleteTablet for 3 replicas of tablet c228b5f32a6f4709a1affb9e17d1b892
I20260924 04:17:41.395750 20002 tablet_service.cc:1514] Processing DeleteTablet for tablet c228b5f32a6f4709a1affb9e17d1b892 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:17:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55414
I20260924 04:17:41.395750 19734 tablet_service.cc:1514] Processing DeleteTablet for tablet c228b5f32a6f4709a1affb9e17d1b892 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:17:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39994
I20260924 04:17:41.396268 20110 tablet_replica.cc:330] T c228b5f32a6f4709a1affb9e17d1b892 P 22033ce134ca48acb4494a579cedc26c: stopping tablet replica
I20260924 04:17:41.396270 19868 tablet_service.cc:1514] Processing DeleteTablet for tablet c228b5f32a6f4709a1affb9e17d1b892 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:17:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39626
I20260924 04:17:41.396516 20109 tablet_replica.cc:330] T c228b5f32a6f4709a1affb9e17d1b892 P ad83e16c91c34404a8e16cc4705b069a: stopping tablet replica
I20260924 04:17:41.396701 20111 tablet_replica.cc:330] T c228b5f32a6f4709a1affb9e17d1b892 P 7675d8f361574350ad626141be4594d9: stopping tablet replica
I20260924 04:17:41.396855 20109 raft_consensus.cc:2229] T c228b5f32a6f4709a1affb9e17d1b892 P ad83e16c91c34404a8e16cc4705b069a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:17:41.397017 20111 raft_consensus.cc:2229] T c228b5f32a6f4709a1affb9e17d1b892 P 7675d8f361574350ad626141be4594d9 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:17:41.397185 20109 raft_consensus.cc:2258] T c228b5f32a6f4709a1affb9e17d1b892 P ad83e16c91c34404a8e16cc4705b069a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:17:41.397490 20111 raft_consensus.cc:2258] T c228b5f32a6f4709a1affb9e17d1b892 P 7675d8f361574350ad626141be4594d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:17:41.397511 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 19671
I20260924 04:17:41.397643 20110 raft_consensus.cc:2229] T c228b5f32a6f4709a1affb9e17d1b892 P 22033ce134ca48acb4494a579cedc26c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:17:41.399879 20111 ts_tablet_manager.cc:1905] T c228b5f32a6f4709a1affb9e17d1b892 P 7675d8f361574350ad626141be4594d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:17:41.402128 20109 ts_tablet_manager.cc:1905] T c228b5f32a6f4709a1affb9e17d1b892 P ad83e16c91c34404a8e16cc4705b069a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:17:41.402798 20111 ts_tablet_manager.cc:1918] T c228b5f32a6f4709a1affb9e17d1b892 P 7675d8f361574350ad626141be4594d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:17:41.402958 20111 log.cc:1196] T c228b5f32a6f4709a1affb9e17d1b892 P 7675d8f361574350ad626141be4594d9: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/ts-1/wal/wals/c228b5f32a6f4709a1affb9e17d1b892
I20260924 04:17:41.403340 20111 ts_tablet_manager.cc:1939] T c228b5f32a6f4709a1affb9e17d1b892 P 7675d8f361574350ad626141be4594d9: Deleting consensus metadata
I20260924 04:17:41.404628 19594 catalog_manager.cc:4841] TS 7675d8f361574350ad626141be4594d9 (127.0.98.2:42143): tablet c228b5f32a6f4709a1affb9e17d1b892 (table test-table [id=584837b57bf94df585db0a10047ee8e5]) successfully deleted
I20260924 04:17:41.404726 20109 ts_tablet_manager.cc:1918] T c228b5f32a6f4709a1affb9e17d1b892 P ad83e16c91c34404a8e16cc4705b069a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:17:41.404808 20109 log.cc:1196] T c228b5f32a6f4709a1affb9e17d1b892 P ad83e16c91c34404a8e16cc4705b069a: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790223216265354-392-0/minicluster-data/ts-2/wal/wals/c228b5f32a6f4709a1affb9e17d1b892
I20260924 04:17:41.405089 20109 ts_tablet_manager.cc:1939] T c228b5f32a6f4709a1affb9e17d1b892 P ad83e16c91c34404a8e16cc4705b069a: Deleting consensus metadata
I20260924 04:17:41.406250 19593 catalog_manager.cc:4841] TS ad83e16c91c34404a8e16cc4705b069a (127.0.98.3:46183): tablet c228b5f32a6f4709a1affb9e17d1b892 (table test-table [id=584837b57bf94df585db0a10047ee8e5]) successfully deleted
W20260924 04:17:41.406309 19596 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44465 (error 108)
I20260924 04:17:41.406526 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 19805
I20260924 04:17:41.407728 20083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:41.407472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44465 (local address 127.0.0.1:40000)
0924 04:17:41.407583 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:41.407651 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 04:17:41.407869 19596 catalog_manager.cc:4584] TS 22033ce134ca48acb4494a579cedc26c (127.0.98.1:44465): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c228b5f32a6f4709a1affb9e17d1b892: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44465: connect: Connection refused (error 111)
I20260924 04:17:41.411659 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 19939
I20260924 04:17:41.416416 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 19577
2026-09-24T04:17:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2367 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-24T04:17:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:17:41Z Disabled control of system clock
I20260924 04:17:41.444718 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37267
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37267
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 04:17:41.553252 20116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:17:41.553539 20116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:17:41.553579 20116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:17:41.557627 20116 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:17:41.557686 20116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:17:41.557704 20116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:17:41.557722 20116 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:17:41.557739 20116 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:17:41.562465 20116 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37267
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:17:41.563465 20116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:17:41.564738 20116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:17:41.572654 20122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:17:41.572757 20121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:17:41.572654 20124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:17:41.572736 20116 server_base.cc:1034] running on GCE node
I20260924 04:17:41.573429 20116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:17:41.574427 20116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:17:41.575578 20116 hybrid_clock.cc:648] HybridClock initialized: now 1790223461575561 us; error 35 us; skew 500 ppm
I20260924 04:17:41.577246 20116 webserver.cc:460] Webserver started at http://127.0.98.62:38201/ using document root <none> and password file <none>
I20260924 04:17:41.577767 20116 fs_manager.cc:359] Metadata directory not provided
I20260924 04:17:41.577831 20116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:17:41.578001 20116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:17:41.579950 20116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/master-0/data/instance:
uuid: "ce04128e73c0415184e07f857f975ec3"
format_stamp: "Formatted at 2026-09-24 04:17:41 on dist-test-slave-p401"
server_key: "a18616e40e1eb4dafa1e6e1f773ec8d0"
server_key_iv: "fd186172e3fb789ef95b8363822c0d0f"
server_key_version: "encryptionkey@0"
I20260924 04:17:41.580459 20116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/master-0/wal/instance:
uuid: "ce04128e73c0415184e07f857f975ec3"
format_stamp: "Formatted at 2026-09-24 04:17:41 on dist-test-slave-p401"
server_key: "a18616e40e1eb4dafa1e6e1f773ec8d0"
server_key_iv: "fd186172e3fb789ef95b8363822c0d0f"
server_key_version: "encryptionkey@0"
I20260924 04:17:41.583768 20116 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:17:41.585978 20130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:41.587210 20116 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:17:41.587347 20116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/master-0/wal
uuid: "ce04128e73c0415184e07f857f975ec3"
format_stamp: "Formatted at 2026-09-24 04:17:41 on dist-test-slave-p401"
server_key: "a18616e40e1eb4dafa1e6e1f773ec8d0"
server_key_iv: "fd186172e3fb789ef95b8363822c0d0f"
server_key_version: "encryptionkey@0"
I20260924 04:17:41.587436 20116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:17:41.597872 20116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:17:41.598440 20116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:17:41.598587 20116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:17:41.604755 20116 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37267
I20260924 04:17:41.604782 20182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37267 every 8 connection(s)
I20260924 04:17:41.605609 20116 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:17:41.608521 20183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:17:41.609875 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 20116
I20260924 04:17:41.609990 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/master-0/wal/instance
I20260924 04:17:41.610255 392 external_mini_cluster.cc:1428] Setting key 8bac3cce24349ef0d03444355d14e2fa
I20260924 04:17:41.614864 20183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce04128e73c0415184e07f857f975ec3: Bootstrap starting.
I20260924 04:17:41.617265 20183 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce04128e73c0415184e07f857f975ec3: Neither blocks nor log segments found. Creating new log.
I20260924 04:17:41.617985 20183 log.cc:824] T 00000000000000000000000000000000 P ce04128e73c0415184e07f857f975ec3: Log is configured to *not* fsync() on all Append() calls
I20260924 04:17:41.619997 20186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:41.611590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32926 (local address 127.0.98.62:37267)
0924 04:17:41.612224 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:41.612243 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:41.612285 (+ 42us) server_negotiation.cc:407] Connection header received
0924 04:17:41.613082 (+ 797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:41.613106 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:41.613586 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:41.613963 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:17:41.614375 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:41.615790 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:41.616615 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:41.617275 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:41.617427 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:41.618120 (+ 693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:17:41.618125 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:17:41.618140 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:17:41.618557 (+ 417us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:17:41.618608 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:17:41.618632 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:17:41.618733 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:41.618863 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 04:17:41.619109 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":103,"threads_started":1}
I20260924 04:17:41.620054 20183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce04128e73c0415184e07f857f975ec3: No bootstrap required, opened a new log
I20260924 04:17:41.623701 20183 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce04128e73c0415184e07f857f975ec3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce04128e73c0415184e07f857f975ec3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37267 } }
I20260924 04:17:41.623901 20183 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce04128e73c0415184e07f857f975ec3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:17:41.623939 20183 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce04128e73c0415184e07f857f975ec3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce04128e73c0415184e07f857f975ec3, State: Initialized, Role: FOLLOWER
I20260924 04:17:41.624307 20183 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce04128e73c0415184e07f857f975ec3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce04128e73c0415184e07f857f975ec3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37267 } }
I20260924 04:17:41.624481 20183 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce04128e73c0415184e07f857f975ec3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:17:41.624544 20183 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce04128e73c0415184e07f857f975ec3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:17:41.624639 20183 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce04128e73c0415184e07f857f975ec3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:17:41.625492 20183 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce04128e73c0415184e07f857f975ec3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce04128e73c0415184e07f857f975ec3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37267 } }
I20260924 04:17:41.625792 20183 leader_election.cc:302] T 00000000000000000000000000000000 P ce04128e73c0415184e07f857f975ec3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce04128e73c0415184e07f857f975ec3; no voters:
I20260924 04:17:41.626041 20183 leader_election.cc:288] T 00000000000000000000000000000000 P ce04128e73c0415184e07f857f975ec3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:17:41.626123 20188 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce04128e73c0415184e07f857f975ec3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:17:41.626353 20188 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce04128e73c0415184e07f857f975ec3 [term 1 LEADER]: Becoming Leader. State: Replica: ce04128e73c0415184e07f857f975ec3, State: Running, Role: LEADER
I20260924 04:17:41.626694 20188 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce04128e73c0415184e07f857f975ec3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce04128e73c0415184e07f857f975ec3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37267 } }
I20260924 04:17:41.627204 20183 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce04128e73c0415184e07f857f975ec3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:17:41.628396 20189 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce04128e73c0415184e07f857f975ec3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce04128e73c0415184e07f857f975ec3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce04128e73c0415184e07f857f975ec3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37267 } } }
I20260924 04:17:41.628468 20190 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce04128e73c0415184e07f857f975ec3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce04128e73c0415184e07f857f975ec3. Latest consensus state: current_term: 1 leader_uuid: "ce04128e73c0415184e07f857f975ec3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce04128e73c0415184e07f857f975ec3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37267 } } }
I20260924 04:17:41.628609 20189 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce04128e73c0415184e07f857f975ec3 [sys.catalog]: This master's current role is: LEADER
I20260924 04:17:41.628613 20190 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce04128e73c0415184e07f857f975ec3 [sys.catalog]: This master's current role is: LEADER
W20260924 04:17:41.631603 20203 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ce04128e73c0415184e07f857f975ec3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:17:41.631729 20203 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:17:41.631860 20200 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:17:41.632599 20200 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:17:41.637240 20200 catalog_manager.cc:1347] Generated new cluster ID: 16760ec12353429589edf01ed96fed6c
I20260924 04:17:41.637300 20200 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:17:41.651933 20200 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:17:41.653085 20200 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:17:41.662804 20200 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce04128e73c0415184e07f857f975ec3: Generated new TSK 0
I20260924 04:17:41.663633 20200 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:17:41.673429 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37267
--builtin_ntp_servers=127.0.98.20:44695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 04:17:41.780444 20207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:17:41.780714 20207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:17:41.780752 20207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:17:41.784770 20207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:17:41.784832 20207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:17:41.784888 20207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:17:41.789768 20207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:17:41.790803 20207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:17:41.792235 20207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:17:41.801307 20215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:17:41.801339 20212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:17:41.801434 20207 server_base.cc:1034] running on GCE node
W20260924 04:17:41.801334 20213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:17:41.801873 20207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:17:41.802464 20207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:17:41.803617 20207 hybrid_clock.cc:648] HybridClock initialized: now 1790223461803595 us; error 35 us; skew 500 ppm
I20260924 04:17:41.805354 20207 webserver.cc:460] Webserver started at http://127.0.98.1:40617/ using document root <none> and password file <none>
I20260924 04:17:41.805879 20207 fs_manager.cc:359] Metadata directory not provided
I20260924 04:17:41.805941 20207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:17:41.806116 20207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:17:41.808068 20207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/ts-0/data/instance:
uuid: "9060bfabb0c64cb582b6504793360d76"
format_stamp: "Formatted at 2026-09-24 04:17:41 on dist-test-slave-p401"
server_key: "04277286d41625a8fa836ac8b668c757"
server_key_iv: "9733ae1b90804ac2f7f9074f4cc93d18"
server_key_version: "encryptionkey@0"
I20260924 04:17:41.808598 20207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9060bfabb0c64cb582b6504793360d76"
format_stamp: "Formatted at 2026-09-24 04:17:41 on dist-test-slave-p401"
server_key: "04277286d41625a8fa836ac8b668c757"
server_key_iv: "9733ae1b90804ac2f7f9074f4cc93d18"
server_key_version: "encryptionkey@0"
I20260924 04:17:41.811998 20207 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:17:41.814188 20221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:41.815366 20207 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:17:41.815464 20207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/ts-0/wal
uuid: "9060bfabb0c64cb582b6504793360d76"
format_stamp: "Formatted at 2026-09-24 04:17:41 on dist-test-slave-p401"
server_key: "04277286d41625a8fa836ac8b668c757"
server_key_iv: "9733ae1b90804ac2f7f9074f4cc93d18"
server_key_version: "encryptionkey@0"
I20260924 04:17:41.815550 20207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:17:41.833520 20207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:17:41.834142 20207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:17:41.834312 20207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:17:41.834849 20207 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:17:41.835741 20207 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:17:41.835788 20207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:41.835839 20207 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:17:41.835865 20207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:41.844264 20207 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41583
I20260924 04:17:41.844339 20334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41583 every 8 connection(s)
I20260924 04:17:41.845116 20207 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:17:41.849403 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 20207
I20260924 04:17:41.849509 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:17:41.849761 392 external_mini_cluster.cc:1428] Setting key 2e0d58acfe3c0f82d0a940e29c42ed7d
I20260924 04:17:41.851517 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37267
--builtin_ntp_servers=127.0.98.20:44695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 04:17:41.854599 20186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:41.846757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57823 (local address 127.0.98.62:37267)
0924 04:17:41.846881 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:41.846885 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:41.847954 (+ 1069us) server_negotiation.cc:407] Connection header received
0924 04:17:41.849325 (+ 1371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:41.849330 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:41.849393 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:41.849480 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:17:41.850551 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:41.851092 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:41.852668 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:41.852900 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:41.853002 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:41.854132 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:17:41.854135 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:17:41.854140 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:17:41.854151 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:17:41.854171 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:17:41.854178 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:17:41.854237 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:41.854398 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 04:17:41.854459 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 04:17:41.855441 20337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:41.847140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37267 (local address 127.0.98.1:57823)
0924 04:17:41.847806 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:41.847847 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 04:17:41.849115 (+ 1268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:17:41.849658 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:17:41.849665 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:17:41.849758 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:17:41.850398 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:41.850417 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:41.851217 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:41.851222 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:41.852545 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:41.852557 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:41.853046 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:41.853050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:41.853229 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:41.853839 (+ 610us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:17:41.853841 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:17:41.853881 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:17:41.853942 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:17:41.854252 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:17:41.854261 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:17:41.854267 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 04:17:41.854383 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 04:17:41.854670 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":104,"threads_started":1}
I20260924 04:17:41.856781 20335 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37267
I20260924 04:17:41.857003 20335 heartbeater.cc:464] Registering TS with master...
I20260924 04:17:41.857542 20335 heartbeater.cc:511] Master 127.0.98.62:37267 requested a full tablet report, sending...
I20260924 04:17:41.859179 20147 ts_manager.cc:194] Registered new tserver with Master: 9060bfabb0c64cb582b6504793360d76 (127.0.98.1:41583)
I20260924 04:17:41.860790 20147 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:57823
W20260924 04:17:41.957377 20338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:17:41.957624 20338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:17:41.957661 20338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:17:41.961671 20338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:17:41.961728 20338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:17:41.961782 20338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:17:41.966465 20338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:17:41.967536 20338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:17:41.968871 20338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:17:41.978024 20344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:17:41.978039 20346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:17:41.978034 20343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:17:41.978339 20338 server_base.cc:1034] running on GCE node
I20260924 04:17:41.978760 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:17:41.979454 20338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:17:41.980633 20338 hybrid_clock.cc:648] HybridClock initialized: now 1790223461980593 us; error 49 us; skew 500 ppm
I20260924 04:17:41.982496 20338 webserver.cc:460] Webserver started at http://127.0.98.2:44769/ using document root <none> and password file <none>
I20260924 04:17:41.983048 20338 fs_manager.cc:359] Metadata directory not provided
I20260924 04:17:41.983129 20338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:17:41.983353 20338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:17:41.985388 20338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/ts-1/data/instance:
uuid: "ba366cfe8fea48119d56e6b2bdc92781"
format_stamp: "Formatted at 2026-09-24 04:17:41 on dist-test-slave-p401"
server_key: "8f52c15d4261f6412acf9fcc6b1fd13b"
server_key_iv: "8a3f3f1b219b8915ba5fabd1e4524289"
server_key_version: "encryptionkey@0"
I20260924 04:17:41.985911 20338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ba366cfe8fea48119d56e6b2bdc92781"
format_stamp: "Formatted at 2026-09-24 04:17:41 on dist-test-slave-p401"
server_key: "8f52c15d4261f6412acf9fcc6b1fd13b"
server_key_iv: "8a3f3f1b219b8915ba5fabd1e4524289"
server_key_version: "encryptionkey@0"
I20260924 04:17:41.989539 20338 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260924 04:17:41.991994 20352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:41.993151 20338 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 04:17:41.993273 20338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/ts-1/wal
uuid: "ba366cfe8fea48119d56e6b2bdc92781"
format_stamp: "Formatted at 2026-09-24 04:17:41 on dist-test-slave-p401"
server_key: "8f52c15d4261f6412acf9fcc6b1fd13b"
server_key_iv: "8a3f3f1b219b8915ba5fabd1e4524289"
server_key_version: "encryptionkey@0"
I20260924 04:17:41.993386 20338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:17:42.030380 20338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:17:42.031112 20338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:17:42.031318 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:17:42.031967 20338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:17:42.032948 20338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:17:42.032996 20338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:42.033051 20338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:17:42.033080 20338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:42.041226 20338 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41597
I20260924 04:17:42.041256 20465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41597 every 8 connection(s)
I20260924 04:17:42.042100 20338 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:17:42.047768 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 20338
I20260924 04:17:42.047894 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:17:42.048195 392 external_mini_cluster.cc:1428] Setting key a578eb77684bdc6b00e5b5e64135fb11
I20260924 04:17:42.050403 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37267
--builtin_ntp_servers=127.0.98.20:44695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 04:17:42.051604 20186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:42.043827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51071 (local address 127.0.98.62:37267)
0924 04:17:42.043942 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:42.043946 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:42.044979 (+ 1033us) server_negotiation.cc:407] Connection header received
0924 04:17:42.046317 (+ 1338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:42.046321 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:42.046378 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:42.046449 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:17:42.047355 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:42.048064 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:42.049587 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:42.049877 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:42.049985 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:42.051123 (+ 1138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:17:42.051127 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:17:42.051129 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:17:42.051142 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:17:42.051164 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:17:42.051175 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:17:42.051275 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:42.051402 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 04:17:42.051461 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 04:17:42.052735 20468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:42.044204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37267 (local address 127.0.98.2:51071)
0924 04:17:42.044831 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:42.044871 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 04:17:42.046113 (+ 1242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:17:42.046629 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:17:42.046636 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:17:42.046724 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:17:42.047203 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:42.047215 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:42.048205 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:42.048208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:42.049444 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:42.049456 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:42.050070 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:42.050074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:42.050226 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:42.050838 (+ 612us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:17:42.050840 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:17:42.050884 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:17:42.050951 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:17:42.051280 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:17:42.051287 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:17:42.051291 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 04:17:42.051394 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 04:17:42.051666 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":101,"threads_started":1}
I20260924 04:17:42.053881 20466 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37267
I20260924 04:17:42.054095 20466 heartbeater.cc:464] Registering TS with master...
I20260924 04:17:42.054590 20466 heartbeater.cc:511] Master 127.0.98.62:37267 requested a full tablet report, sending...
I20260924 04:17:42.055552 20147 ts_manager.cc:194] Registered new tserver with Master: ba366cfe8fea48119d56e6b2bdc92781 (127.0.98.2:41597)
I20260924 04:17:42.056093 20147 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:51071
W20260924 04:17:42.158143 20469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:17:42.158408 20469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:17:42.158459 20469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:17:42.162461 20469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:17:42.162536 20469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:17:42.162612 20469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:17:42.167390 20469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:17:42.168397 20469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:17:42.169752 20469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:17:42.178819 20469 server_base.cc:1034] running on GCE node
W20260924 04:17:42.178782 20474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:17:42.178782 20477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:17:42.178788 20475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:17:42.179397 20469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:17:42.180042 20469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:17:42.181226 20469 hybrid_clock.cc:648] HybridClock initialized: now 1790223462181225 us; error 40 us; skew 500 ppm
I20260924 04:17:42.183050 20469 webserver.cc:460] Webserver started at http://127.0.98.3:36131/ using document root <none> and password file <none>
I20260924 04:17:42.183640 20469 fs_manager.cc:359] Metadata directory not provided
I20260924 04:17:42.183707 20469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:17:42.183904 20469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:17:42.185904 20469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/ts-2/data/instance:
uuid: "85b11710f7c74ff8ab206deb17f30083"
format_stamp: "Formatted at 2026-09-24 04:17:42 on dist-test-slave-p401"
server_key: "46427ffc9a13b561cf1d526997f68dee"
server_key_iv: "3b1e8ef5b157ff026099862415e3af4e"
server_key_version: "encryptionkey@0"
I20260924 04:17:42.186476 20469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/ts-2/wal/instance:
uuid: "85b11710f7c74ff8ab206deb17f30083"
format_stamp: "Formatted at 2026-09-24 04:17:42 on dist-test-slave-p401"
server_key: "46427ffc9a13b561cf1d526997f68dee"
server_key_iv: "3b1e8ef5b157ff026099862415e3af4e"
server_key_version: "encryptionkey@0"
I20260924 04:17:42.189999 20469 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260924 04:17:42.192330 20483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:42.193457 20469 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:17:42.193567 20469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/ts-2/wal
uuid: "85b11710f7c74ff8ab206deb17f30083"
format_stamp: "Formatted at 2026-09-24 04:17:42 on dist-test-slave-p401"
server_key: "46427ffc9a13b561cf1d526997f68dee"
server_key_iv: "3b1e8ef5b157ff026099862415e3af4e"
server_key_version: "encryptionkey@0"
I20260924 04:17:42.193691 20469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:17:42.215365 20469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:17:42.216046 20469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:17:42.216214 20469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:17:42.216835 20469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:17:42.217751 20469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:17:42.217797 20469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:42.217849 20469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:17:42.217877 20469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:42.226325 20469 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34095
I20260924 04:17:42.226369 20596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34095 every 8 connection(s)
I20260924 04:17:42.227273 20469 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:17:42.236161 20186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:42.229087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36113 (local address 127.0.98.62:37267)
0924 04:17:42.229221 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:42.229224 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:42.230321 (+ 1097us) server_negotiation.cc:407] Connection header received
0924 04:17:42.231678 (+ 1357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:42.231681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:42.231727 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:42.231815 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:17:42.232728 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:42.233289 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:42.234385 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:42.234616 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:42.234696 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:42.235752 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:17:42.235755 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:17:42.235757 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:17:42.235769 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:17:42.235789 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:17:42.235797 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:17:42.235860 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:42.235960 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 04:17:42.236014 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 04:17:42.236975 20599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:42.229517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37267 (local address 127.0.98.3:36113)
0924 04:17:42.230174 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:42.230215 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 04:17:42.231457 (+ 1242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:17:42.232004 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:17:42.232011 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:17:42.232106 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:17:42.232591 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:42.232604 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:42.233391 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:42.233393 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:42.234274 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:42.234283 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:42.234751 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:42.234754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:42.234905 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:42.235509 (+ 604us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:17:42.235510 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:17:42.235550 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:17:42.235612 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:17:42.235864 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:17:42.235868 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:17:42.235871 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 04:17:42.235954 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 04:17:42.236196 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":105,"threads_started":1}
I20260924 04:17:42.237207 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 20469
I20260924 04:17:42.237313 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:17:42.237571 392 external_mini_cluster.cc:1428] Setting key 6c6855d6b0399f4be5377843bddca7c4
I20260924 04:17:42.238114 20597 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37267
I20260924 04:17:42.238351 20597 heartbeater.cc:464] Registering TS with master...
I20260924 04:17:42.238965 20597 heartbeater.cc:511] Master 127.0.98.62:37267 requested a full tablet report, sending...
I20260924 04:17:42.239954 20147 ts_manager.cc:194] Registered new tserver with Master: 85b11710f7c74ff8ab206deb17f30083 (127.0.98.3:34095)
I20260924 04:17:42.240491 20147 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:36113
I20260924 04:17:42.251608 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:17:42.257153 20186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:42.253508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32934 (local address 127.0.98.62:37267)
0924 04:17:42.253645 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:42.253649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:42.253776 (+ 127us) server_negotiation.cc:407] Connection header received
0924 04:17:42.253888 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:42.253890 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:42.253931 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:42.253997 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:17:42.254225 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:42.254720 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:42.255611 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:42.255843 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:42.255918 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:42.256739 (+ 821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:17:42.256742 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:17:42.256744 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:17:42.256767 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:17:42.256783 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:17:42.256790 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:17:42.256842 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:42.256984 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 04:17:42.257033 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 04:17:42.260082 20147 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32934:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:17:42.263026 20147 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:17:42.276552 20611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:42.271811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41583 (local address 127.0.0.1:34954)
0924 04:17:42.272356 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:42.272368 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:17:42.272440 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:17:42.272876 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:17:42.272879 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:17:42.272894 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:17:42.272983 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:42.272989 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:42.274037 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:42.274040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:42.275130 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:42.275139 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:42.276012 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:42.276015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:42.276087 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:42.276188 (+ 101us) client_negotiation.cc:769] Sending connection context
0924 04:17:42.276267 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 04:17:42.276400 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":52,"thread_start_us":87,"threads_started":1}
I20260924 04:17:42.276636 20610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:42.271467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34954 (local address 127.0.98.1:41583)
0924 04:17:42.272497 (+ 1030us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:42.272508 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:42.272528 (+ 20us) server_negotiation.cc:407] Connection header received
0924 04:17:42.272599 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:42.272604 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:42.272756 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:42.272874 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:17:42.273090 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:42.273929 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:42.275262 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:42.275898 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:42.276181 (+ 283us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:42.276360 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:42.276401 (+ 41us) server_negotiation.cc:299] Negotiation successful
0924 04:17:42.276473 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":933,"thread_start_us":116,"threads_started":1}
I20260924 04:17:42.276762 20612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:42.271662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41597 (local address 127.0.0.1:54082)
0924 04:17:42.272103 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:42.272116 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:17:42.272195 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:17:42.273132 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:17:42.273135 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:17:42.273153 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:17:42.273227 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:42.273233 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:42.274423 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:42.274427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:42.275485 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:42.275494 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:42.276437 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:42.276441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:42.276523 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:42.276543 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 04:17:42.276587 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 04:17:42.276644 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"mutex_wait_us":1,"thread_start_us":115,"threads_started":1}
I20260924 04:17:42.277154 20608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:42.270972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34095 (local address 127.0.0.1:45008)
0924 04:17:42.271320 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:42.271365 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 04:17:42.271479 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:17:42.272328 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:17:42.272333 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:17:42.272356 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:17:42.272480 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:42.272486 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:42.273822 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:42.273826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:42.274999 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:42.275009 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:42.275955 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:42.275959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:42.276075 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:42.276172 (+ 97us) client_negotiation.cc:769] Sending connection context
0924 04:17:42.276274 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 04:17:42.276926 (+ 652us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":582,"thread_start_us":75,"threads_started":1}
I20260924 04:17:42.276636 20609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:42.271131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45008 (local address 127.0.98.3:34095)
0924 04:17:42.271644 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:42.271654 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:42.271678 (+ 24us) server_negotiation.cc:407] Connection header received
0924 04:17:42.271755 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:42.271760 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:42.272000 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:42.272155 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:17:42.272601 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:42.273715 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:42.275140 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:42.275842 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:42.275942 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:42.276116 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:42.276376 (+ 260us) server_negotiation.cc:299] Negotiation successful
0924 04:17:42.276473 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":198,"threads_started":1}
I20260924 04:17:42.277366 20613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:42.271980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54082 (local address 127.0.98.2:41597)
0924 04:17:42.272658 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:42.272670 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:42.272710 (+ 40us) server_negotiation.cc:407] Connection header received
0924 04:17:42.272781 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:42.272786 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:42.272969 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:42.273138 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:17:42.273329 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:42.274311 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:42.275610 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:42.276321 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:42.276922 (+ 601us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:42.277094 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:42.277135 (+ 41us) server_negotiation.cc:299] Negotiation successful
0924 04:17:42.277214 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":106,"threads_started":1}
I20260924 04:17:42.279718 20400 tablet_service.cc:1467] Processing CreateTablet for tablet ef26e1d538b444c09d3a5844353e2b5d (DEFAULT_TABLE table=test-table [id=19c9b637d1364f4f952e9456397a464c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:17:42.279757 20269 tablet_service.cc:1467] Processing CreateTablet for tablet ef26e1d538b444c09d3a5844353e2b5d (DEFAULT_TABLE table=test-table [id=19c9b637d1364f4f952e9456397a464c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:17:42.279923 20531 tablet_service.cc:1467] Processing CreateTablet for tablet ef26e1d538b444c09d3a5844353e2b5d (DEFAULT_TABLE table=test-table [id=19c9b637d1364f4f952e9456397a464c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:17:42.280637 20400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef26e1d538b444c09d3a5844353e2b5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:17:42.280741 20531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef26e1d538b444c09d3a5844353e2b5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:17:42.280741 20269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef26e1d538b444c09d3a5844353e2b5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:17:42.286936 20614 tablet_bootstrap.cc:492] T ef26e1d538b444c09d3a5844353e2b5d P ba366cfe8fea48119d56e6b2bdc92781: Bootstrap starting.
I20260924 04:17:42.287914 20616 tablet_bootstrap.cc:492] T ef26e1d538b444c09d3a5844353e2b5d P 85b11710f7c74ff8ab206deb17f30083: Bootstrap starting.
I20260924 04:17:42.288232 20615 tablet_bootstrap.cc:492] T ef26e1d538b444c09d3a5844353e2b5d P 9060bfabb0c64cb582b6504793360d76: Bootstrap starting.
I20260924 04:17:42.289145 20614 tablet_bootstrap.cc:654] T ef26e1d538b444c09d3a5844353e2b5d P ba366cfe8fea48119d56e6b2bdc92781: Neither blocks nor log segments found. Creating new log.
I20260924 04:17:42.289850 20615 tablet_bootstrap.cc:654] T ef26e1d538b444c09d3a5844353e2b5d P 9060bfabb0c64cb582b6504793360d76: Neither blocks nor log segments found. Creating new log.
I20260924 04:17:42.289980 20614 log.cc:824] T ef26e1d538b444c09d3a5844353e2b5d P ba366cfe8fea48119d56e6b2bdc92781: Log is configured to *not* fsync() on all Append() calls
I20260924 04:17:42.290040 20616 tablet_bootstrap.cc:654] T ef26e1d538b444c09d3a5844353e2b5d P 85b11710f7c74ff8ab206deb17f30083: Neither blocks nor log segments found. Creating new log.
I20260924 04:17:42.290608 20615 log.cc:824] T ef26e1d538b444c09d3a5844353e2b5d P 9060bfabb0c64cb582b6504793360d76: Log is configured to *not* fsync() on all Append() calls
I20260924 04:17:42.290733 20616 log.cc:824] T ef26e1d538b444c09d3a5844353e2b5d P 85b11710f7c74ff8ab206deb17f30083: Log is configured to *not* fsync() on all Append() calls
I20260924 04:17:42.292114 20614 tablet_bootstrap.cc:492] T ef26e1d538b444c09d3a5844353e2b5d P ba366cfe8fea48119d56e6b2bdc92781: No bootstrap required, opened a new log
I20260924 04:17:42.292301 20614 ts_tablet_manager.cc:1397] T ef26e1d538b444c09d3a5844353e2b5d P ba366cfe8fea48119d56e6b2bdc92781: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:17:42.292580 20615 tablet_bootstrap.cc:492] T ef26e1d538b444c09d3a5844353e2b5d P 9060bfabb0c64cb582b6504793360d76: No bootstrap required, opened a new log
I20260924 04:17:42.292691 20616 tablet_bootstrap.cc:492] T ef26e1d538b444c09d3a5844353e2b5d P 85b11710f7c74ff8ab206deb17f30083: No bootstrap required, opened a new log
I20260924 04:17:42.292764 20615 ts_tablet_manager.cc:1397] T ef26e1d538b444c09d3a5844353e2b5d P 9060bfabb0c64cb582b6504793360d76: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 04:17:42.292863 20616 ts_tablet_manager.cc:1397] T ef26e1d538b444c09d3a5844353e2b5d P 85b11710f7c74ff8ab206deb17f30083: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 04:17:42.296036 20616 raft_consensus.cc:348] T ef26e1d538b444c09d3a5844353e2b5d P 85b11710f7c74ff8ab206deb17f30083 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85b11710f7c74ff8ab206deb17f30083" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34095 } } peers { permanent_uuid: "ba366cfe8fea48119d56e6b2bdc92781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41597 } } peers { permanent_uuid: "9060bfabb0c64cb582b6504793360d76" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41583 } }
I20260924 04:17:42.296331 20616 raft_consensus.cc:374] T ef26e1d538b444c09d3a5844353e2b5d P 85b11710f7c74ff8ab206deb17f30083 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:17:42.296382 20616 raft_consensus.cc:729] T ef26e1d538b444c09d3a5844353e2b5d P 85b11710f7c74ff8ab206deb17f30083 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85b11710f7c74ff8ab206deb17f30083, State: Initialized, Role: FOLLOWER
I20260924 04:17:42.296608 20614 raft_consensus.cc:348] T ef26e1d538b444c09d3a5844353e2b5d P ba366cfe8fea48119d56e6b2bdc92781 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85b11710f7c74ff8ab206deb17f30083" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34095 } } peers { permanent_uuid: "ba366cfe8fea48119d56e6b2bdc92781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41597 } } peers { permanent_uuid: "9060bfabb0c64cb582b6504793360d76" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41583 } }
I20260924 04:17:42.296605 20615 raft_consensus.cc:348] T ef26e1d538b444c09d3a5844353e2b5d P 9060bfabb0c64cb582b6504793360d76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85b11710f7c74ff8ab206deb17f30083" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34095 } } peers { permanent_uuid: "ba366cfe8fea48119d56e6b2bdc92781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41597 } } peers { permanent_uuid: "9060bfabb0c64cb582b6504793360d76" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41583 } }
I20260924 04:17:42.296844 20615 raft_consensus.cc:374] T ef26e1d538b444c09d3a5844353e2b5d P 9060bfabb0c64cb582b6504793360d76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:17:42.296844 20614 raft_consensus.cc:374] T ef26e1d538b444c09d3a5844353e2b5d P ba366cfe8fea48119d56e6b2bdc92781 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:17:42.296785 20616 consensus_queue.cc:260] T ef26e1d538b444c09d3a5844353e2b5d P 85b11710f7c74ff8ab206deb17f30083 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85b11710f7c74ff8ab206deb17f30083" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34095 } } peers { permanent_uuid: "ba366cfe8fea48119d56e6b2bdc92781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41597 } } peers { permanent_uuid: "9060bfabb0c64cb582b6504793360d76" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41583 } }
I20260924 04:17:42.296896 20614 raft_consensus.cc:729] T ef26e1d538b444c09d3a5844353e2b5d P ba366cfe8fea48119d56e6b2bdc92781 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba366cfe8fea48119d56e6b2bdc92781, State: Initialized, Role: FOLLOWER
I20260924 04:17:42.296896 20615 raft_consensus.cc:729] T ef26e1d538b444c09d3a5844353e2b5d P 9060bfabb0c64cb582b6504793360d76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9060bfabb0c64cb582b6504793360d76, State: Initialized, Role: FOLLOWER
I20260924 04:17:42.297375 20614 consensus_queue.cc:260] T ef26e1d538b444c09d3a5844353e2b5d P ba366cfe8fea48119d56e6b2bdc92781 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85b11710f7c74ff8ab206deb17f30083" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34095 } } peers { permanent_uuid: "ba366cfe8fea48119d56e6b2bdc92781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41597 } } peers { permanent_uuid: "9060bfabb0c64cb582b6504793360d76" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41583 } }
I20260924 04:17:42.297524 20597 heartbeater.cc:503] Master 127.0.98.62:37267 was elected leader, sending a full tablet report...
I20260924 04:17:42.297801 20616 ts_tablet_manager.cc:1428] T ef26e1d538b444c09d3a5844353e2b5d P 85b11710f7c74ff8ab206deb17f30083: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 04:17:42.298142 20466 heartbeater.cc:503] Master 127.0.98.62:37267 was elected leader, sending a full tablet report...
I20260924 04:17:42.298364 20614 ts_tablet_manager.cc:1428] T ef26e1d538b444c09d3a5844353e2b5d P ba366cfe8fea48119d56e6b2bdc92781: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 04:17:42.297345 20615 consensus_queue.cc:260] T ef26e1d538b444c09d3a5844353e2b5d P 9060bfabb0c64cb582b6504793360d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85b11710f7c74ff8ab206deb17f30083" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34095 } } peers { permanent_uuid: "ba366cfe8fea48119d56e6b2bdc92781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41597 } } peers { permanent_uuid: "9060bfabb0c64cb582b6504793360d76" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41583 } }
I20260924 04:17:42.299396 20615 ts_tablet_manager.cc:1428] T ef26e1d538b444c09d3a5844353e2b5d P 9060bfabb0c64cb582b6504793360d76: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260924 04:17:42.299685 20335 heartbeater.cc:503] Master 127.0.98.62:37267 was elected leader, sending a full tablet report...
W20260924 04:17:42.346179 20336 tablet.cc:2369] T ef26e1d538b444c09d3a5844353e2b5d P 9060bfabb0c64cb582b6504793360d76: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:17:42.355406 20622 raft_consensus.cc:482] T ef26e1d538b444c09d3a5844353e2b5d P 9060bfabb0c64cb582b6504793360d76 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:17:42.355597 20622 raft_consensus.cc:504] T ef26e1d538b444c09d3a5844353e2b5d P 9060bfabb0c64cb582b6504793360d76 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85b11710f7c74ff8ab206deb17f30083" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34095 } } peers { permanent_uuid: "ba366cfe8fea48119d56e6b2bdc92781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41597 } } peers { permanent_uuid: "9060bfabb0c64cb582b6504793360d76" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41583 } }
I20260924 04:17:42.356724 20622 leader_election.cc:288] T ef26e1d538b444c09d3a5844353e2b5d P 9060bfabb0c64cb582b6504793360d76 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85b11710f7c74ff8ab206deb17f30083 (127.0.98.3:34095), ba366cfe8fea48119d56e6b2bdc92781 (127.0.98.2:41597)
I20260924 04:17:42.361289 20613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:42.357446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57963 (local address 127.0.98.2:41597)
0924 04:17:42.357566 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:42.357571 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:42.357649 (+ 78us) server_negotiation.cc:407] Connection header received
0924 04:17:42.357794 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:42.357796 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:42.357839 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:42.357909 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:17:42.358110 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:42.358801 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:42.360145 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:42.360731 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:42.360834 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:42.360892 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:42.361099 (+ 207us) server_negotiation.cc:299] Negotiation successful
0924 04:17:42.361163 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 04:17:42.361289 20609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:42.357058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49369 (local address 127.0.98.3:34095)
0924 04:17:42.357204 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:42.357209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:42.357289 (+ 80us) server_negotiation.cc:407] Connection header received
0924 04:17:42.357448 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:42.357451 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:42.357502 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:42.357584 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:17:42.357868 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:42.358613 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:42.360145 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:42.360735 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:42.360834 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:42.360892 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:42.361099 (+ 207us) server_negotiation.cc:299] Negotiation successful
0924 04:17:42.361165 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 04:17:42.361370 20623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:42.356937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34095 (local address 127.0.98.1:49369)
0924 04:17:42.357197 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:42.357212 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:17:42.357335 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:17:42.357611 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:17:42.357614 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:17:42.357634 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:17:42.357770 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:42.357777 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:42.358726 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:42.358729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:42.360015 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:42.360026 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:42.360841 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:42.360844 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:42.360961 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:42.361007 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 04:17:42.361079 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 04:17:42.361206 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":27,"spinlock_wait_cycles":7424,"thread_start_us":100,"threads_started":1}
I20260924 04:17:42.361370 20624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:42.357301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41597 (local address 127.0.98.1:57963)
0924 04:17:42.357572 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:42.357585 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:17:42.357682 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:17:42.357922 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:17:42.357924 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:17:42.357939 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:17:42.358022 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:42.358026 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:42.358889 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:42.358893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:42.360019 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:17:42.360026 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:17:42.360839 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:17:42.360843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:17:42.361001 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:42.361018 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 04:17:42.361079 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 04:17:42.361192 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":84,"spinlock_wait_cycles":2048,"thread_start_us":63,"threads_started":1}
I20260924 04:17:42.362021 20420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef26e1d538b444c09d3a5844353e2b5d" candidate_uuid: "9060bfabb0c64cb582b6504793360d76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba366cfe8fea48119d56e6b2bdc92781" is_pre_election: true
I20260924 04:17:42.362021 20551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef26e1d538b444c09d3a5844353e2b5d" candidate_uuid: "9060bfabb0c64cb582b6504793360d76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85b11710f7c74ff8ab206deb17f30083" is_pre_election: true
I20260924 04:17:42.362334 20420 raft_consensus.cc:2454] T ef26e1d538b444c09d3a5844353e2b5d P ba366cfe8fea48119d56e6b2bdc92781 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9060bfabb0c64cb582b6504793360d76 in term 0.
I20260924 04:17:42.362334 20551 raft_consensus.cc:2454] T ef26e1d538b444c09d3a5844353e2b5d P 85b11710f7c74ff8ab206deb17f30083 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9060bfabb0c64cb582b6504793360d76 in term 0.
I20260924 04:17:42.362776 20222 leader_election.cc:302] T ef26e1d538b444c09d3a5844353e2b5d P 9060bfabb0c64cb582b6504793360d76 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9060bfabb0c64cb582b6504793360d76, ba366cfe8fea48119d56e6b2bdc92781; no voters:
I20260924 04:17:42.363008 20622 raft_consensus.cc:2781] T ef26e1d538b444c09d3a5844353e2b5d P 9060bfabb0c64cb582b6504793360d76 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:17:42.363091 20622 raft_consensus.cc:482] T ef26e1d538b444c09d3a5844353e2b5d P 9060bfabb0c64cb582b6504793360d76 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:17:42.363138 20622 raft_consensus.cc:3037] T ef26e1d538b444c09d3a5844353e2b5d P 9060bfabb0c64cb582b6504793360d76 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:17:42.364116 20622 raft_consensus.cc:504] T ef26e1d538b444c09d3a5844353e2b5d P 9060bfabb0c64cb582b6504793360d76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85b11710f7c74ff8ab206deb17f30083" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34095 } } peers { permanent_uuid: "ba366cfe8fea48119d56e6b2bdc92781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41597 } } peers { permanent_uuid: "9060bfabb0c64cb582b6504793360d76" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41583 } }
I20260924 04:17:42.364410 20622 leader_election.cc:288] T ef26e1d538b444c09d3a5844353e2b5d P 9060bfabb0c64cb582b6504793360d76 [CANDIDATE]: Term 1 election: Requested vote from peers 85b11710f7c74ff8ab206deb17f30083 (127.0.98.3:34095), ba366cfe8fea48119d56e6b2bdc92781 (127.0.98.2:41597)
I20260924 04:17:42.364739 20551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef26e1d538b444c09d3a5844353e2b5d" candidate_uuid: "9060bfabb0c64cb582b6504793360d76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85b11710f7c74ff8ab206deb17f30083"
I20260924 04:17:42.364763 20420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef26e1d538b444c09d3a5844353e2b5d" candidate_uuid: "9060bfabb0c64cb582b6504793360d76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba366cfe8fea48119d56e6b2bdc92781"
I20260924 04:17:42.364864 20420 raft_consensus.cc:3037] T ef26e1d538b444c09d3a5844353e2b5d P ba366cfe8fea48119d56e6b2bdc92781 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:17:42.364864 20551 raft_consensus.cc:3037] T ef26e1d538b444c09d3a5844353e2b5d P 85b11710f7c74ff8ab206deb17f30083 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:17:42.366056 20551 raft_consensus.cc:2454] T ef26e1d538b444c09d3a5844353e2b5d P 85b11710f7c74ff8ab206deb17f30083 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9060bfabb0c64cb582b6504793360d76 in term 1.
I20260924 04:17:42.366056 20420 raft_consensus.cc:2454] T ef26e1d538b444c09d3a5844353e2b5d P ba366cfe8fea48119d56e6b2bdc92781 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9060bfabb0c64cb582b6504793360d76 in term 1.
I20260924 04:17:42.366333 20222 leader_election.cc:302] T ef26e1d538b444c09d3a5844353e2b5d P 9060bfabb0c64cb582b6504793360d76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85b11710f7c74ff8ab206deb17f30083, 9060bfabb0c64cb582b6504793360d76; no voters:
I20260924 04:17:42.366497 20622 raft_consensus.cc:2781] T ef26e1d538b444c09d3a5844353e2b5d P 9060bfabb0c64cb582b6504793360d76 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:17:42.366729 20622 raft_consensus.cc:686] T ef26e1d538b444c09d3a5844353e2b5d P 9060bfabb0c64cb582b6504793360d76 [term 1 LEADER]: Becoming Leader. State: Replica: 9060bfabb0c64cb582b6504793360d76, State: Running, Role: LEADER
I20260924 04:17:42.367339 20622 consensus_queue.cc:237] T ef26e1d538b444c09d3a5844353e2b5d P 9060bfabb0c64cb582b6504793360d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85b11710f7c74ff8ab206deb17f30083" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34095 } } peers { permanent_uuid: "ba366cfe8fea48119d56e6b2bdc92781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41597 } } peers { permanent_uuid: "9060bfabb0c64cb582b6504793360d76" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41583 } }
I20260924 04:17:42.370432 20144 catalog_manager.cc:5495] T ef26e1d538b444c09d3a5844353e2b5d P 9060bfabb0c64cb582b6504793360d76 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9060bfabb0c64cb582b6504793360d76 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "9060bfabb0c64cb582b6504793360d76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85b11710f7c74ff8ab206deb17f30083" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba366cfe8fea48119d56e6b2bdc92781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9060bfabb0c64cb582b6504793360d76" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41583 } health_report { overall_health: HEALTHY } } }
I20260924 04:17:42.399739 20610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:42.396132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34966 (local address 127.0.98.1:41583)
0924 04:17:42.396275 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:42.396279 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:42.396294 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:17:42.396414 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:42.396417 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:42.396488 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:42.396594 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:17:42.396854 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:42.397383 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:42.398290 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:42.398515 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:42.398610 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:42.398896 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:17:42.399346 (+ 450us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:17:42.399456 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:42.399544 (+ 88us) server_negotiation.cc:299] Negotiation successful
0924 04:17:42.399604 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:17:42.405876 20420 raft_consensus.cc:1261] T ef26e1d538b444c09d3a5844353e2b5d P ba366cfe8fea48119d56e6b2bdc92781 [term 1 FOLLOWER]: Refusing update from remote peer 9060bfabb0c64cb582b6504793360d76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:17:42.405876 20551 raft_consensus.cc:1261] T ef26e1d538b444c09d3a5844353e2b5d P 85b11710f7c74ff8ab206deb17f30083 [term 1 FOLLOWER]: Refusing update from remote peer 9060bfabb0c64cb582b6504793360d76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:17:42.406739 20625 consensus_queue.cc:1035] T ef26e1d538b444c09d3a5844353e2b5d P 9060bfabb0c64cb582b6504793360d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85b11710f7c74ff8ab206deb17f30083" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:17:42.407018 20622 consensus_queue.cc:1035] T ef26e1d538b444c09d3a5844353e2b5d P 9060bfabb0c64cb582b6504793360d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba366cfe8fea48119d56e6b2bdc92781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:17:42.425768 20146 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:32934:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:17:42.425966 20146 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:32934:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:17:42.428565 20146 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ce04128e73c0415184e07f857f975ec3: Sending DeleteTablet for 3 replicas of tablet ef26e1d538b444c09d3a5844353e2b5d
I20260924 04:17:42.429452 20400 tablet_service.cc:1514] Processing DeleteTablet for tablet ef26e1d538b444c09d3a5844353e2b5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:17:42 UTC) from {username='slave'} at 127.0.0.1:54082
I20260924 04:17:42.429543 20269 tablet_service.cc:1514] Processing DeleteTablet for tablet ef26e1d538b444c09d3a5844353e2b5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:17:42 UTC) from {username='slave'} at 127.0.0.1:34954
I20260924 04:17:42.429926 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 20207
I20260924 04:17:42.429963 20531 tablet_service.cc:1514] Processing DeleteTablet for tablet ef26e1d538b444c09d3a5844353e2b5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:17:42 UTC) from {username='slave'} at 127.0.0.1:45008
I20260924 04:17:42.430546 20638 tablet_replica.cc:330] T ef26e1d538b444c09d3a5844353e2b5d P ba366cfe8fea48119d56e6b2bdc92781: stopping tablet replica
I20260924 04:17:42.431090 20640 tablet_replica.cc:330] T ef26e1d538b444c09d3a5844353e2b5d P 85b11710f7c74ff8ab206deb17f30083: stopping tablet replica
I20260924 04:17:42.431185 20638 raft_consensus.cc:2229] T ef26e1d538b444c09d3a5844353e2b5d P ba366cfe8fea48119d56e6b2bdc92781 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:17:42.431394 20640 raft_consensus.cc:2229] T ef26e1d538b444c09d3a5844353e2b5d P 85b11710f7c74ff8ab206deb17f30083 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:17:42.431543 20638 raft_consensus.cc:2258] T ef26e1d538b444c09d3a5844353e2b5d P ba366cfe8fea48119d56e6b2bdc92781 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:17:42.431666 20640 raft_consensus.cc:2258] T ef26e1d538b444c09d3a5844353e2b5d P 85b11710f7c74ff8ab206deb17f30083 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:17:42.432947 20640 ts_tablet_manager.cc:1905] T ef26e1d538b444c09d3a5844353e2b5d P 85b11710f7c74ff8ab206deb17f30083: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:17:42.432957 20638 ts_tablet_manager.cc:1905] T ef26e1d538b444c09d3a5844353e2b5d P ba366cfe8fea48119d56e6b2bdc92781: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:17:42.435022 20640 ts_tablet_manager.cc:1918] T ef26e1d538b444c09d3a5844353e2b5d P 85b11710f7c74ff8ab206deb17f30083: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:17:42.435117 20640 log.cc:1196] T ef26e1d538b444c09d3a5844353e2b5d P 85b11710f7c74ff8ab206deb17f30083: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/ts-2/wal/wals/ef26e1d538b444c09d3a5844353e2b5d
I20260924 04:17:42.435381 20640 ts_tablet_manager.cc:1939] T ef26e1d538b444c09d3a5844353e2b5d P 85b11710f7c74ff8ab206deb17f30083: Deleting consensus metadata
I20260924 04:17:42.435498 20638 ts_tablet_manager.cc:1918] T ef26e1d538b444c09d3a5844353e2b5d P ba366cfe8fea48119d56e6b2bdc92781: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:17:42.435577 20638 log.cc:1196] T ef26e1d538b444c09d3a5844353e2b5d P ba366cfe8fea48119d56e6b2bdc92781: Deleting WAL directory at /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790223216265354-392-0/minicluster-data/ts-1/wal/wals/ef26e1d538b444c09d3a5844353e2b5d
I20260924 04:17:42.435855 20638 ts_tablet_manager.cc:1939] T ef26e1d538b444c09d3a5844353e2b5d P ba366cfe8fea48119d56e6b2bdc92781: Deleting consensus metadata
I20260924 04:17:42.436741 20131 catalog_manager.cc:4841] TS 85b11710f7c74ff8ab206deb17f30083 (127.0.98.3:34095): tablet ef26e1d538b444c09d3a5844353e2b5d (table test-table [id=19c9b637d1364f4f952e9456397a464c]) successfully deleted
I20260924 04:17:42.437155 20131 catalog_manager.cc:4841] TS ba366cfe8fea48119d56e6b2bdc92781 (127.0.98.2:41597): tablet ef26e1d538b444c09d3a5844353e2b5d (table test-table [id=19c9b637d1364f4f952e9456397a464c]) successfully deleted
W20260924 04:17:42.437875 20132 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41583 (error 108)
I20260924 04:17:42.438051 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 20338
I20260924 04:17:42.440552 20608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:42.440288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41583 (local address 127.0.0.1:34978)
0924 04:17:42.440408 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 04:17:42.440476 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 04:17:42.440794 20132 catalog_manager.cc:4584] TS 9060bfabb0c64cb582b6504793360d76 (127.0.98.1:41583): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ef26e1d538b444c09d3a5844353e2b5d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41583: connect: Connection refused (error 111)
I20260924 04:17:42.444708 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 20469
I20260924 04:17:42.449467 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 20116
2026-09-24T04:17:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1032 ms)
[----------] 6 tests from EncryptionPolicyTest (9662 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790223216265354-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:17:42 dist-test-slave-p401 krb5kdc[20644](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:17:42 dist-test-slave-p401 krb5kdc[20644](info): set up 2 sockets
Sep 24 04:17:42 dist-test-slave-p401 krb5kdc[20644](info): commencing operation
krb5kdc: starting...
W20260924 04:17:43.403139 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790223216265354-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790223216265354-392-0/krb5kdc/test-user.keytab.
Sep 24 04:17:43 dist-test-slave-p401 krb5kdc[20644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:17:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:17:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790223216265354-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790223216265354-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:17:43.518469 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790223216265354-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790223216265354-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790223216265354-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34407
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790223216265354-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34407
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:17:43.622539 20657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:17:43.622779 20657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:17:43.622819 20657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:17:43.626746 20657 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:17:43.626808 20657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:17:43.626825 20657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:17:43.626840 20657 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:17:43.626853 20657 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:17:43.632026 20657 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790223216265354-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790223216265354-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34407
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790223216265354-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790223216265354-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:17:43.632949 20657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:17:43.633911 20657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:17:43.641848 20665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:17:43.641878 20662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:17:43.641973 20657 server_base.cc:1034] running on GCE node
W20260924 04:17:43.641883 20663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:17:43.642659 20657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:17:43.643766 20657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:17:43.644919 20657 hybrid_clock.cc:648] HybridClock initialized: now 1790223463644903 us; error 27 us; skew 500 ppm
Sep 24 04:17:43 dist-test-slave-p401 krb5kdc[20644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:17:43.650657 20657 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:17:43.651566 20657 webserver.cc:460] Webserver started at http://127.0.0.1:35661/ using document root <none> and password file <none>
I20260924 04:17:43.652122 20657 fs_manager.cc:359] Metadata directory not provided
I20260924 04:17:43.652184 20657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:17:43.652396 20657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:17:43.654417 20657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790223216265354-392-0/minicluster-data/master-0/data/instance:
uuid: "accfb8e234aa43d5955c4b9c850403c6"
format_stamp: "Formatted at 2026-09-24 04:17:43 on dist-test-slave-p401"
server_key: "6e73c9e7c193fda0979bda35af803eb1"
server_key_iv: "faf1789776890730e85dba5bf34f1d1c"
server_key_version: "encryptionkey@0"
I20260924 04:17:43.654954 20657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790223216265354-392-0/minicluster-data/master-0/wal/instance:
uuid: "accfb8e234aa43d5955c4b9c850403c6"
format_stamp: "Formatted at 2026-09-24 04:17:43 on dist-test-slave-p401"
server_key: "6e73c9e7c193fda0979bda35af803eb1"
server_key_iv: "faf1789776890730e85dba5bf34f1d1c"
server_key_version: "encryptionkey@0"
I20260924 04:17:43.658571 20657 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 04:17:43.660907 20672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:43.662057 20657 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:17:43.662163 20657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790223216265354-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790223216265354-392-0/minicluster-data/master-0/wal
uuid: "accfb8e234aa43d5955c4b9c850403c6"
format_stamp: "Formatted at 2026-09-24 04:17:43 on dist-test-slave-p401"
server_key: "6e73c9e7c193fda0979bda35af803eb1"
server_key_iv: "faf1789776890730e85dba5bf34f1d1c"
server_key_version: "encryptionkey@0"
I20260924 04:17:43.662269 20657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790223216265354-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790223216265354-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790223216265354-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:17:43.673257 20657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:17:43.676033 20657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:17:43.676169 20657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:17:43.682446 20657 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34407
I20260924 04:17:43.682488 20724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34407 every 8 connection(s)
I20260924 04:17:43.683350 20657 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:17:43.683646 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 20657
I20260924 04:17:43.683770 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790223216265354-392-0/minicluster-data/master-0/wal/instance
I20260924 04:17:43.684060 392 external_mini_cluster.cc:1428] Setting key 4459e3cdebb9d78abdb1f01f85aa149b
I20260924 04:17:43.686213 20725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:17:43.692414 20725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P accfb8e234aa43d5955c4b9c850403c6: Bootstrap starting.
Sep 24 04:17:43 dist-test-slave-p401 krb5kdc[20644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 04:17:43.694494 20725 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P accfb8e234aa43d5955c4b9c850403c6: Neither blocks nor log segments found. Creating new log.
I20260924 04:17:43.695144 20725 log.cc:824] T 00000000000000000000000000000000 P accfb8e234aa43d5955c4b9c850403c6: Log is configured to *not* fsync() on all Append() calls
I20260924 04:17:43.697165 20725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P accfb8e234aa43d5955c4b9c850403c6: No bootstrap required, opened a new log
I20260924 04:17:43.698150 20728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:43.685489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56775 (local address 127.0.0.1:34407)
0924 04:17:43.685883 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:43.685902 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:43.685937 (+ 35us) server_negotiation.cc:407] Connection header received
0924 04:17:43.686578 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:43.686595 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:43.686937 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:43.687211 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:17:43.688074 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:43.689049 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:43.690174 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:43.690655 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:43.690772 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:43.693438 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:17:43.693443 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:17:43.693455 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:17:43.693481 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:17:43.695023 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:17:43.695554 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:17:43.695559 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:17:43.695563 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:17:43.695663 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:17:43.695996 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:17:43.695999 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:17:43.696001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:17:43.696476 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:17:43.696746 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:43.697049 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 04:17:43.697271 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":116,"threads_started":1}
I20260924 04:17:43.700443 20725 raft_consensus.cc:348] T 00000000000000000000000000000000 P accfb8e234aa43d5955c4b9c850403c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "accfb8e234aa43d5955c4b9c850403c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34407 } }
I20260924 04:17:43.700683 20725 raft_consensus.cc:374] T 00000000000000000000000000000000 P accfb8e234aa43d5955c4b9c850403c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:17:43.700752 20725 raft_consensus.cc:729] T 00000000000000000000000000000000 P accfb8e234aa43d5955c4b9c850403c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: accfb8e234aa43d5955c4b9c850403c6, State: Initialized, Role: FOLLOWER
I20260924 04:17:43.701139 20725 consensus_queue.cc:260] T 00000000000000000000000000000000 P accfb8e234aa43d5955c4b9c850403c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "accfb8e234aa43d5955c4b9c850403c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34407 } }
I20260924 04:17:43.701288 20725 raft_consensus.cc:388] T 00000000000000000000000000000000 P accfb8e234aa43d5955c4b9c850403c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:17:43.701354 20725 raft_consensus.cc:482] T 00000000000000000000000000000000 P accfb8e234aa43d5955c4b9c850403c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:17:43.701445 20725 raft_consensus.cc:3037] T 00000000000000000000000000000000 P accfb8e234aa43d5955c4b9c850403c6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:17:43.701505 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:17:43.702479 20725 raft_consensus.cc:504] T 00000000000000000000000000000000 P accfb8e234aa43d5955c4b9c850403c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "accfb8e234aa43d5955c4b9c850403c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34407 } }
I20260924 04:17:43.702841 20725 leader_election.cc:302] T 00000000000000000000000000000000 P accfb8e234aa43d5955c4b9c850403c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: accfb8e234aa43d5955c4b9c850403c6; no voters:
I20260924 04:17:43.703150 20725 leader_election.cc:288] T 00000000000000000000000000000000 P accfb8e234aa43d5955c4b9c850403c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:17:43.703225 20734 raft_consensus.cc:2781] T 00000000000000000000000000000000 P accfb8e234aa43d5955c4b9c850403c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:17:43.703472 20734 raft_consensus.cc:686] T 00000000000000000000000000000000 P accfb8e234aa43d5955c4b9c850403c6 [term 1 LEADER]: Becoming Leader. State: Replica: accfb8e234aa43d5955c4b9c850403c6, State: Running, Role: LEADER
I20260924 04:17:43.703766 20734 consensus_queue.cc:237] T 00000000000000000000000000000000 P accfb8e234aa43d5955c4b9c850403c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "accfb8e234aa43d5955c4b9c850403c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34407 } }
I20260924 04:17:43.704381 20725 sys_catalog.cc:564] T 00000000000000000000000000000000 P accfb8e234aa43d5955c4b9c850403c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:17:43.704828 20735 sys_catalog.cc:455] T 00000000000000000000000000000000 P accfb8e234aa43d5955c4b9c850403c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader accfb8e234aa43d5955c4b9c850403c6. Latest consensus state: current_term: 1 leader_uuid: "accfb8e234aa43d5955c4b9c850403c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "accfb8e234aa43d5955c4b9c850403c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34407 } } }
I20260924 04:17:43.704949 20735 sys_catalog.cc:458] T 00000000000000000000000000000000 P accfb8e234aa43d5955c4b9c850403c6 [sys.catalog]: This master's current role is: LEADER
I20260924 04:17:43.705667 20736 sys_catalog.cc:455] T 00000000000000000000000000000000 P accfb8e234aa43d5955c4b9c850403c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "accfb8e234aa43d5955c4b9c850403c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "accfb8e234aa43d5955c4b9c850403c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34407 } } }
I20260924 04:17:43.705798 20736 sys_catalog.cc:458] T 00000000000000000000000000000000 P accfb8e234aa43d5955c4b9c850403c6 [sys.catalog]: This master's current role is: LEADER
I20260924 04:17:43.706147 20741 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:17:43.709622 20741 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:17:43.712644 20728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:43.703926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32999 (local address 127.0.0.1:34407)
0924 04:17:43.704019 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:43.704023 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:43.704036 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:17:43.704193 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:43.704196 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:43.704249 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:43.704294 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:17:43.705135 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:43.705902 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:43.707130 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:43.707486 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:43.707636 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:43.709078 (+ 1442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:17:43.709083 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:17:43.709086 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:17:43.709112 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:17:43.710752 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:17:43.711312 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:17:43.711316 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:17:43.711318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:17:43.711409 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:17:43.711761 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:17:43.711765 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:17:43.711766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:17:43.711959 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:17:43.712096 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:43.712383 (+ 287us) server_negotiation.cc:299] Negotiation successful
0924 04:17:43.712471 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 04:17:43.715401 20741 catalog_manager.cc:1347] Generated new cluster ID: 680a15efd2c94f96a7df596e85245582
I20260924 04:17:43.715454 20741 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:17:43.732200 20741 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:17:43.733395 20741 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:17:43.745324 20741 catalog_manager.cc:5868] T 00000000000000000000000000000000 P accfb8e234aa43d5955c4b9c850403c6: Generated new TSK 0
I20260924 04:17:43.745959 20741 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:17:43.754199 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 20657
W20260924 04:17:43.758724 20642 connection.cc:535] client connection to 127.0.0.1:34407 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34407): Connection reset by peer (error 104)
2026-09-24T04:17:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1303 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790223216265354-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:17:43 dist-test-slave-p401 krb5kdc[20756](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:17:43 dist-test-slave-p401 krb5kdc[20756](info): set up 2 sockets
Sep 24 04:17:43 dist-test-slave-p401 krb5kdc[20756](info): commencing operation
krb5kdc: starting...
W20260924 04:17:44.699086 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790223216265354-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790223216265354-392-0/krb5kdc/test-user.keytab.
Sep 24 04:17:44 dist-test-slave-p401 krb5kdc[20756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:17:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:17:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790223216265354-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790223216265354-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:17:44.814052 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790223216265354-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790223216265354-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790223216265354-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46643
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790223216265354-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46643
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:17:44.919284 20769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:17:44.919536 20769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:17:44.919593 20769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:17:44.923533 20769 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:17:44.923591 20769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:17:44.923616 20769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:17:44.923647 20769 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:17:44.923674 20769 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:17:44.928866 20769 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790223216265354-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790223216265354-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46643
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790223216265354-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20769
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790223216265354-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:17:44.929849 20769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:17:44.930816 20769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:17:44.939044 20769 server_base.cc:1034] running on GCE node
W20260924 04:17:44.938992 20774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:17:44.938969 20775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:17:44.939002 20777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:17:44.939786 20769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:17:44.940802 20769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:17:44.941948 20769 hybrid_clock.cc:648] HybridClock initialized: now 1790223464941928 us; error 33 us; skew 500 ppm
Sep 24 04:17:44 dist-test-slave-p401 krb5kdc[20756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223464, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:17:44.944514 20769 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:17:44.945345 20769 webserver.cc:460] Webserver started at http://127.0.0.1:45607/ using document root <none> and password file <none>
I20260924 04:17:44.945875 20769 fs_manager.cc:359] Metadata directory not provided
I20260924 04:17:44.945938 20769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:17:44.946132 20769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:17:44.948110 20769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790223216265354-392-0/minicluster-data/master-0/data/instance:
uuid: "0d63d6cb16614f2aaa4e4df01799c459"
format_stamp: "Formatted at 2026-09-24 04:17:44 on dist-test-slave-p401"
server_key: "1143ff0738b9cbb93132a13ed57a7302"
server_key_iv: "df422c067992fda41f4ced0c9ea0f5d9"
server_key_version: "encryptionkey@0"
I20260924 04:17:44.948617 20769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790223216265354-392-0/minicluster-data/master-0/wal/instance:
uuid: "0d63d6cb16614f2aaa4e4df01799c459"
format_stamp: "Formatted at 2026-09-24 04:17:44 on dist-test-slave-p401"
server_key: "1143ff0738b9cbb93132a13ed57a7302"
server_key_iv: "df422c067992fda41f4ced0c9ea0f5d9"
server_key_version: "encryptionkey@0"
I20260924 04:17:44.952096 20769 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 04:17:44.954273 20784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:44.955446 20769 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:17:44.955547 20769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790223216265354-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790223216265354-392-0/minicluster-data/master-0/wal
uuid: "0d63d6cb16614f2aaa4e4df01799c459"
format_stamp: "Formatted at 2026-09-24 04:17:44 on dist-test-slave-p401"
server_key: "1143ff0738b9cbb93132a13ed57a7302"
server_key_iv: "df422c067992fda41f4ced0c9ea0f5d9"
server_key_version: "encryptionkey@0"
I20260924 04:17:44.955637 20769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790223216265354-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790223216265354-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790223216265354-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:17:44.980692 20769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:17:44.983605 20769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:17:44.983784 20769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:17:44.990478 20769 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46643
I20260924 04:17:44.990496 20836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46643 every 8 connection(s)
I20260924 04:17:44.991508 20769 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:17:44.994285 20837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:17:44.999785 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 20769
I20260924 04:17:44.999787 20837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d63d6cb16614f2aaa4e4df01799c459: Bootstrap starting.
I20260924 04:17:44.999922 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790223216265354-392-0/minicluster-data/master-0/wal/instance
I20260924 04:17:45.000212 392 external_mini_cluster.cc:1428] Setting key 3b69d52d1293e1931b188b14ff505928
I20260924 04:17:45.002326 20837 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d63d6cb16614f2aaa4e4df01799c459: Neither blocks nor log segments found. Creating new log.
I20260924 04:17:45.003011 20837 log.cc:824] T 00000000000000000000000000000000 P 0d63d6cb16614f2aaa4e4df01799c459: Log is configured to *not* fsync() on all Append() calls
I20260924 04:17:45.004748 20837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d63d6cb16614f2aaa4e4df01799c459: No bootstrap required, opened a new log
I20260924 04:17:45.008309 20837 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d63d6cb16614f2aaa4e4df01799c459 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d63d6cb16614f2aaa4e4df01799c459" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46643 } }
I20260924 04:17:45.008543 20837 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d63d6cb16614f2aaa4e4df01799c459 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:17:45.008598 20837 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d63d6cb16614f2aaa4e4df01799c459 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d63d6cb16614f2aaa4e4df01799c459, State: Initialized, Role: FOLLOWER
I20260924 04:17:45.008692 20840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:45.001479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39077 (local address 127.0.0.1:46643)
0924 04:17:45.001869 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:45.001889 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:45.001923 (+ 34us) server_negotiation.cc:407] Connection header received
0924 04:17:45.002579 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:45.002598 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:45.002943 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:45.003264 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:17:45.003670 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:45.004622 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:45.005499 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:45.005959 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:45.006100 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:45.006979 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:17:45.006984 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:17:45.006996 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:17:45.007424 (+ 428us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:17:45.007482 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:17:45.007505 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:17:45.007616 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:45.007792 (+ 176us) server_negotiation.cc:299] Negotiation successful
0924 04:17:45.007927 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":100,"threads_started":1}
I20260924 04:17:45.009073 20837 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d63d6cb16614f2aaa4e4df01799c459 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d63d6cb16614f2aaa4e4df01799c459" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46643 } }
I20260924 04:17:45.009238 20837 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d63d6cb16614f2aaa4e4df01799c459 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:17:45.009300 20837 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d63d6cb16614f2aaa4e4df01799c459 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:17:45.009389 20837 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d63d6cb16614f2aaa4e4df01799c459 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:17:45.010313 20837 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d63d6cb16614f2aaa4e4df01799c459 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d63d6cb16614f2aaa4e4df01799c459" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46643 } }
I20260924 04:17:45.010658 20837 leader_election.cc:302] T 00000000000000000000000000000000 P 0d63d6cb16614f2aaa4e4df01799c459 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d63d6cb16614f2aaa4e4df01799c459; no voters:
I20260924 04:17:45.010986 20837 leader_election.cc:288] T 00000000000000000000000000000000 P 0d63d6cb16614f2aaa4e4df01799c459 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:17:45.011073 20842 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d63d6cb16614f2aaa4e4df01799c459 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:17:45.011339 20842 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d63d6cb16614f2aaa4e4df01799c459 [term 1 LEADER]: Becoming Leader. State: Replica: 0d63d6cb16614f2aaa4e4df01799c459, State: Running, Role: LEADER
I20260924 04:17:45.011699 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:17:45.011698 20842 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d63d6cb16614f2aaa4e4df01799c459 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d63d6cb16614f2aaa4e4df01799c459" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46643 } }
I20260924 04:17:45.012266 20837 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d63d6cb16614f2aaa4e4df01799c459 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:17:45.013334 20843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d63d6cb16614f2aaa4e4df01799c459 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d63d6cb16614f2aaa4e4df01799c459" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d63d6cb16614f2aaa4e4df01799c459" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46643 } } }
I20260924 04:17:45.013504 20843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d63d6cb16614f2aaa4e4df01799c459 [sys.catalog]: This master's current role is: LEADER
I20260924 04:17:45.013797 20843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d63d6cb16614f2aaa4e4df01799c459 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d63d6cb16614f2aaa4e4df01799c459. Latest consensus state: current_term: 1 leader_uuid: "0d63d6cb16614f2aaa4e4df01799c459" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d63d6cb16614f2aaa4e4df01799c459" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46643 } } }
I20260924 04:17:45.013931 20843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d63d6cb16614f2aaa4e4df01799c459 [sys.catalog]: This master's current role is: LEADER
W20260924 04:17:45.017243 20861 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0d63d6cb16614f2aaa4e4df01799c459: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:17:45.017310 20861 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:17:45.017381 20857 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:17:45.018142 20857 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:17:45.023262 20857 catalog_manager.cc:1347] Generated new cluster ID: 0f5fa127c6804552871ee09f058b65a0
I20260924 04:17:45.023319 20857 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:17:45.025010 20840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:45.013950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50181 (local address 127.0.0.1:46643)
0924 04:17:45.014052 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:45.014056 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:45.014571 (+ 515us) server_negotiation.cc:407] Connection header received
0924 04:17:45.014704 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:45.014707 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:45.014751 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:45.014818 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:17:45.015073 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:45.015644 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:45.016698 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:45.017010 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:45.017114 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:45.024560 (+ 7446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:17:45.024564 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:17:45.024567 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:17:45.024581 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:17:45.024611 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:17:45.024623 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:17:45.024711 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:45.024852 (+ 141us) server_negotiation.cc:299] Negotiation successful
0924 04:17:45.024884 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 04:17:45.063730 20857 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:17:45.064713 20857 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:17:45.073218 20857 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d63d6cb16614f2aaa4e4df01799c459: Generated new TSK 0
I20260924 04:17:45.073958 20857 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:17:45.080029 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 20769
W20260924 04:17:45.084704 20754 connection.cc:535] client connection to 127.0.0.1:46643 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46643): Connection reset by peer (error 104)
2026-09-24T04:17:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1325 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790223216265354-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:17:45 dist-test-slave-p401 krb5kdc[20868](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:17:45 dist-test-slave-p401 krb5kdc[20868](info): set up 2 sockets
Sep 24 04:17:45 dist-test-slave-p401 krb5kdc[20868](info): commencing operation
krb5kdc: starting...
W20260924 04:17:46.027077 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790223216265354-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790223216265354-392-0/krb5kdc/test-user.keytab.
Sep 24 04:17:46 dist-test-slave-p401 krb5kdc[20868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:17:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:17:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790223216265354-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790223216265354-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:17:46.142362 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790223216265354-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790223216265354-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790223216265354-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34871
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790223216265354-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34871
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:17:46.247115 20881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:17:46.247421 20881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:17:46.247478 20881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:17:46.251426 20881 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:17:46.251487 20881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:17:46.251513 20881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:17:46.251533 20881 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:17:46.251554 20881 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:17:46.256767 20881 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790223216265354-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790223216265354-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34871
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790223216265354-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790223216265354-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:17:46.257740 20881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:17:46.258699 20881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:17:46.266646 20887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:17:46.266678 20886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:17:46.266831 20889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:17:46.266741 20881 server_base.cc:1034] running on GCE node
I20260924 04:17:46.267460 20881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:17:46.268518 20881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:17:46.269675 20881 hybrid_clock.cc:648] HybridClock initialized: now 1790223466269650 us; error 36 us; skew 500 ppm
Sep 24 04:17:46 dist-test-slave-p401 krb5kdc[20868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223466, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:17:46.272409 20881 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:17:46.273285 20881 webserver.cc:460] Webserver started at http://127.0.0.1:35153/ using document root <none> and password file <none>
I20260924 04:17:46.273813 20881 fs_manager.cc:359] Metadata directory not provided
I20260924 04:17:46.273880 20881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:17:46.274096 20881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:17:46.276172 20881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790223216265354-392-0/minicluster-data/master-0/data/instance:
uuid: "d6aeae9caa124131a875e607ad9b4669"
format_stamp: "Formatted at 2026-09-24 04:17:46 on dist-test-slave-p401"
server_key: "1a66fde604addf96721e665bb0c8d91b"
server_key_iv: "ccfe8804df52e8948d59f744b030ea0c"
server_key_version: "encryptionkey@0"
I20260924 04:17:46.276748 20881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790223216265354-392-0/minicluster-data/master-0/wal/instance:
uuid: "d6aeae9caa124131a875e607ad9b4669"
format_stamp: "Formatted at 2026-09-24 04:17:46 on dist-test-slave-p401"
server_key: "1a66fde604addf96721e665bb0c8d91b"
server_key_iv: "ccfe8804df52e8948d59f744b030ea0c"
server_key_version: "encryptionkey@0"
I20260924 04:17:46.280486 20881 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:17:46.282742 20896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:46.283944 20881 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:17:46.284051 20881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790223216265354-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790223216265354-392-0/minicluster-data/master-0/wal
uuid: "d6aeae9caa124131a875e607ad9b4669"
format_stamp: "Formatted at 2026-09-24 04:17:46 on dist-test-slave-p401"
server_key: "1a66fde604addf96721e665bb0c8d91b"
server_key_iv: "ccfe8804df52e8948d59f744b030ea0c"
server_key_version: "encryptionkey@0"
I20260924 04:17:46.284173 20881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790223216265354-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790223216265354-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790223216265354-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:17:46.288507 20881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:17:46.291420 20881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:17:46.291568 20881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:17:46.297786 20881 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34871
I20260924 04:17:46.297797 20948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34871 every 8 connection(s)
I20260924 04:17:46.298655 20881 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:17:46.301234 20949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:17:46.306424 20949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6aeae9caa124131a875e607ad9b4669: Bootstrap starting.
I20260924 04:17:46.307890 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 20881
I20260924 04:17:46.308022 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790223216265354-392-0/minicluster-data/master-0/wal/instance
I20260924 04:17:46.308254 392 external_mini_cluster.cc:1428] Setting key 304cd7cc2e87f5bc58344c719ae2f331
I20260924 04:17:46.308789 20949 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6aeae9caa124131a875e607ad9b4669: Neither blocks nor log segments found. Creating new log.
I20260924 04:17:46.309589 20949 log.cc:824] T 00000000000000000000000000000000 P d6aeae9caa124131a875e607ad9b4669: Log is configured to *not* fsync() on all Append() calls
I20260924 04:17:46.311651 20949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6aeae9caa124131a875e607ad9b4669: No bootstrap required, opened a new log
I20260924 04:17:46.313220 20952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:46.309259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59665 (local address 127.0.0.1:34871)
0924 04:17:46.309685 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:46.309705 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:46.309746 (+ 41us) server_negotiation.cc:407] Connection header received
0924 04:17:46.310452 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:46.310473 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:46.310795 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:46.311137 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:17:46.311830 (+ 693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:17:46.311836 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:17:46.311851 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:17:46.311863 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:17:46.311925 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:17:46.311951 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:17:46.312058 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:46.312229 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 04:17:46.312384 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":116,"threads_started":1}
I20260924 04:17:46.315734 20949 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6aeae9caa124131a875e607ad9b4669 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6aeae9caa124131a875e607ad9b4669" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34871 } }
I20260924 04:17:46.315922 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:17:46.315989 20949 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6aeae9caa124131a875e607ad9b4669 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:17:46.316037 20949 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6aeae9caa124131a875e607ad9b4669 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6aeae9caa124131a875e607ad9b4669, State: Initialized, Role: FOLLOWER
I20260924 04:17:46.316455 20949 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6aeae9caa124131a875e607ad9b4669 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6aeae9caa124131a875e607ad9b4669" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34871 } }
I20260924 04:17:46.316597 20949 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6aeae9caa124131a875e607ad9b4669 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:17:46.316655 20949 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6aeae9caa124131a875e607ad9b4669 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:17:46.316744 20949 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6aeae9caa124131a875e607ad9b4669 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:17:46.317750 20949 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6aeae9caa124131a875e607ad9b4669 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6aeae9caa124131a875e607ad9b4669" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34871 } }
I20260924 04:17:46.318084 20949 leader_election.cc:302] T 00000000000000000000000000000000 P d6aeae9caa124131a875e607ad9b4669 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6aeae9caa124131a875e607ad9b4669; no voters:
I20260924 04:17:46.318418 20949 leader_election.cc:288] T 00000000000000000000000000000000 P d6aeae9caa124131a875e607ad9b4669 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:17:46.318498 20959 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6aeae9caa124131a875e607ad9b4669 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:17:46.318751 20959 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6aeae9caa124131a875e607ad9b4669 [term 1 LEADER]: Becoming Leader. State: Replica: d6aeae9caa124131a875e607ad9b4669, State: Running, Role: LEADER
I20260924 04:17:46.319146 20959 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6aeae9caa124131a875e607ad9b4669 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6aeae9caa124131a875e607ad9b4669" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34871 } }
I20260924 04:17:46.319555 20949 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6aeae9caa124131a875e607ad9b4669 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:17:46.319934 20952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:46.318048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46757 (local address 127.0.0.1:34871)
0924 04:17:46.318158 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:46.318161 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:46.318355 (+ 194us) server_negotiation.cc:407] Connection header received
0924 04:17:46.318470 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:46.318473 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:46.318506 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:46.318607 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:17:46.319413 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:17:46.319416 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:17:46.319418 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:17:46.319429 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:17:46.319452 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:17:46.319464 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:17:46.319530 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:46.319799 (+ 269us) server_negotiation.cc:299] Negotiation successful
0924 04:17:46.319835 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 04:17:46.320304 20959 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6aeae9caa124131a875e607ad9b4669 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6aeae9caa124131a875e607ad9b4669. Latest consensus state: current_term: 1 leader_uuid: "d6aeae9caa124131a875e607ad9b4669" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6aeae9caa124131a875e607ad9b4669" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34871 } } }
I20260924 04:17:46.320403 20959 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6aeae9caa124131a875e607ad9b4669 [sys.catalog]: This master's current role is: LEADER
I20260924 04:17:46.320786 20968 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:17:46.321538 20960 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6aeae9caa124131a875e607ad9b4669 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6aeae9caa124131a875e607ad9b4669" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6aeae9caa124131a875e607ad9b4669" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34871 } } }
I20260924 04:17:46.321637 20960 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6aeae9caa124131a875e607ad9b4669 [sys.catalog]: This master's current role is: LEADER
I20260924 04:17:46.323781 20968 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:17:46.328950 20968 catalog_manager.cc:1347] Generated new cluster ID: 487b7518826d4b5aa7aaef5d84c55cef
I20260924 04:17:46.329017 20968 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:17:46.352458 20968 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:17:46.352829 20968 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:17:46.361409 20968 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6aeae9caa124131a875e607ad9b4669: Generated new TSK 0
I20260924 04:17:46.362236 20968 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:17:46.371212 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 20881
2026-09-24T04:17:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1291 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790223216265354-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:17:46 dist-test-slave-p401 krb5kdc[20980](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:17:46 dist-test-slave-p401 krb5kdc[20980](info): set up 2 sockets
Sep 24 04:17:46 dist-test-slave-p401 krb5kdc[20980](info): commencing operation
krb5kdc: starting...
W20260924 04:17:47.319199 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790223216265354-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790223216265354-392-0/krb5kdc/test-user.keytab.
Sep 24 04:17:47 dist-test-slave-p401 krb5kdc[20980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:17:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:17:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790223216265354-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790223216265354-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:17:47.434523 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790223216265354-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790223216265354-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790223216265354-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45105
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790223216265354-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:17:47.543622 20993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:17:47.543883 20993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:17:47.543927 20993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:17:47.547891 20993 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:17:47.547955 20993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:17:47.547971 20993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:17:47.547987 20993 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:17:47.548000 20993 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:17:47.553519 20993 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790223216265354-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790223216265354-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45105
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790223216265354-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790223216265354-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:17:47.554607 20993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:17:47.555619 20993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:17:47.563961 21001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:17:47.563992 20998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:17:47.563979 20999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:17:47.564244 20993 server_base.cc:1034] running on GCE node
I20260924 04:17:47.564874 20993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:17:47.565902 20993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:17:47.567046 20993 hybrid_clock.cc:648] HybridClock initialized: now 1790223467567031 us; error 29 us; skew 500 ppm
Sep 24 04:17:47 dist-test-slave-p401 krb5kdc[20980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223467, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:17:47.569607 20993 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:17:47.570467 20993 webserver.cc:460] Webserver started at http://127.0.0.1:33217/ using document root <none> and password file <none>
I20260924 04:17:47.570984 20993 fs_manager.cc:359] Metadata directory not provided
I20260924 04:17:47.571048 20993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:17:47.571280 20993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:17:47.573243 20993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790223216265354-392-0/minicluster-data/master-0/data/instance:
uuid: "e48276b247114054a0c33162fb19fee4"
format_stamp: "Formatted at 2026-09-24 04:17:47 on dist-test-slave-p401"
server_key: "eab01a38c0c87ab9aad7d384468297cc"
server_key_iv: "7c3faded237b12131bb85f2324b34443"
server_key_version: "encryptionkey@0"
I20260924 04:17:47.573740 20993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790223216265354-392-0/minicluster-data/master-0/wal/instance:
uuid: "e48276b247114054a0c33162fb19fee4"
format_stamp: "Formatted at 2026-09-24 04:17:47 on dist-test-slave-p401"
server_key: "eab01a38c0c87ab9aad7d384468297cc"
server_key_iv: "7c3faded237b12131bb85f2324b34443"
server_key_version: "encryptionkey@0"
I20260924 04:17:47.577193 20993 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260924 04:17:47.579430 21008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:47.580588 20993 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:17:47.580693 20993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790223216265354-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790223216265354-392-0/minicluster-data/master-0/wal
uuid: "e48276b247114054a0c33162fb19fee4"
format_stamp: "Formatted at 2026-09-24 04:17:47 on dist-test-slave-p401"
server_key: "eab01a38c0c87ab9aad7d384468297cc"
server_key_iv: "7c3faded237b12131bb85f2324b34443"
server_key_version: "encryptionkey@0"
I20260924 04:17:47.580794 20993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790223216265354-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790223216265354-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790223216265354-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:17:47.593686 20993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:17:47.596824 20993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:17:47.597003 20993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:17:47.603607 20993 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45105
I20260924 04:17:47.603636 21060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45105 every 8 connection(s)
I20260924 04:17:47.604566 20993 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:17:47.607251 21061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:17:47.610333 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 20993
I20260924 04:17:47.610433 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790223216265354-392-0/minicluster-data/master-0/wal/instance
I20260924 04:17:47.610662 392 external_mini_cluster.cc:1428] Setting key c09a3012eae2509380fdf9ae6ca8bde6
I20260924 04:17:47.612938 21061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e48276b247114054a0c33162fb19fee4: Bootstrap starting.
I20260924 04:17:47.615087 21061 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e48276b247114054a0c33162fb19fee4: Neither blocks nor log segments found. Creating new log.
I20260924 04:17:47.615819 21061 log.cc:824] T 00000000000000000000000000000000 P e48276b247114054a0c33162fb19fee4: Log is configured to *not* fsync() on all Append() calls
I20260924 04:17:47.617794 21061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e48276b247114054a0c33162fb19fee4: No bootstrap required, opened a new log
Sep 24 04:17:47 dist-test-slave-p401 krb5kdc[20980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 04:17:47.621975 21061 raft_consensus.cc:348] T 00000000000000000000000000000000 P e48276b247114054a0c33162fb19fee4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e48276b247114054a0c33162fb19fee4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45105 } }
I20260924 04:17:47.622220 21061 raft_consensus.cc:374] T 00000000000000000000000000000000 P e48276b247114054a0c33162fb19fee4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:17:47.622287 21061 raft_consensus.cc:729] T 00000000000000000000000000000000 P e48276b247114054a0c33162fb19fee4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e48276b247114054a0c33162fb19fee4, State: Initialized, Role: FOLLOWER
I20260924 04:17:47.622797 21061 consensus_queue.cc:260] T 00000000000000000000000000000000 P e48276b247114054a0c33162fb19fee4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e48276b247114054a0c33162fb19fee4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45105 } }
I20260924 04:17:47.622958 21061 raft_consensus.cc:388] T 00000000000000000000000000000000 P e48276b247114054a0c33162fb19fee4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:17:47.623020 21061 raft_consensus.cc:482] T 00000000000000000000000000000000 P e48276b247114054a0c33162fb19fee4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:17:47.623112 21061 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e48276b247114054a0c33162fb19fee4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:17:47.624244 21061 raft_consensus.cc:504] T 00000000000000000000000000000000 P e48276b247114054a0c33162fb19fee4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e48276b247114054a0c33162fb19fee4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45105 } }
I20260924 04:17:47.624615 21061 leader_election.cc:302] T 00000000000000000000000000000000 P e48276b247114054a0c33162fb19fee4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e48276b247114054a0c33162fb19fee4; no voters:
I20260924 04:17:47.624949 21061 leader_election.cc:288] T 00000000000000000000000000000000 P e48276b247114054a0c33162fb19fee4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:17:47.624974 21064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:47.612020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60279 (local address 127.0.0.1:45105)
0924 04:17:47.612451 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:47.612466 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:47.612500 (+ 34us) server_negotiation.cc:407] Connection header received
0924 04:17:47.613128 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:47.613146 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:47.613507 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:47.613823 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:17:47.614590 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:47.615621 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:47.616615 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:47.617157 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:47.617302 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:47.619907 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:17:47.619913 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:17:47.619931 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:17:47.619961 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:17:47.621857 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:17:47.622250 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:17:47.622255 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:17:47.622262 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:17:47.622405 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:17:47.622640 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:17:47.622643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:17:47.622645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:17:47.623180 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:17:47.623429 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:47.623625 (+ 196us) server_negotiation.cc:299] Negotiation successful
0924 04:17:47.623923 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":128,"threads_started":1}
I20260924 04:17:47.625047 21066 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e48276b247114054a0c33162fb19fee4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:17:47.625243 21066 raft_consensus.cc:686] T 00000000000000000000000000000000 P e48276b247114054a0c33162fb19fee4 [term 1 LEADER]: Becoming Leader. State: Replica: e48276b247114054a0c33162fb19fee4, State: Running, Role: LEADER
I20260924 04:17:47.625592 21066 consensus_queue.cc:237] T 00000000000000000000000000000000 P e48276b247114054a0c33162fb19fee4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e48276b247114054a0c33162fb19fee4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45105 } }
I20260924 04:17:47.626248 21061 sys_catalog.cc:564] T 00000000000000000000000000000000 P e48276b247114054a0c33162fb19fee4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:17:47.626756 21067 sys_catalog.cc:455] T 00000000000000000000000000000000 P e48276b247114054a0c33162fb19fee4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e48276b247114054a0c33162fb19fee4. Latest consensus state: current_term: 1 leader_uuid: "e48276b247114054a0c33162fb19fee4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e48276b247114054a0c33162fb19fee4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45105 } } }
I20260924 04:17:47.626878 21067 sys_catalog.cc:458] T 00000000000000000000000000000000 P e48276b247114054a0c33162fb19fee4 [sys.catalog]: This master's current role is: LEADER
I20260924 04:17:47.627501 21068 sys_catalog.cc:455] T 00000000000000000000000000000000 P e48276b247114054a0c33162fb19fee4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e48276b247114054a0c33162fb19fee4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e48276b247114054a0c33162fb19fee4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45105 } } }
I20260924 04:17:47.627588 21068 sys_catalog.cc:458] T 00000000000000000000000000000000 P e48276b247114054a0c33162fb19fee4 [sys.catalog]: This master's current role is: LEADER
I20260924 04:17:47.628242 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260924 04:17:47.631613 21082 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e48276b247114054a0c33162fb19fee4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:17:47.631716 21082 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:17:47.631776 21071 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:17:47.632472 21071 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:17:47.638236 21071 catalog_manager.cc:1347] Generated new cluster ID: b955d78a483d4b908ad56fc4a2634695
I20260924 04:17:47.638300 21071 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:17:47.641959 21064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:47.630483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53367 (local address 127.0.0.1:45105)
0924 04:17:47.630589 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:47.630594 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:47.634303 (+ 3709us) server_negotiation.cc:407] Connection header received
0924 04:17:47.634422 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:47.634425 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:47.634469 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:47.634532 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:17:47.635269 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:47.635814 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:47.636769 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:47.637100 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:47.637215 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:47.638796 (+ 1581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:17:47.638799 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:17:47.638802 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:17:47.638828 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:17:47.640454 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:17:47.640888 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:17:47.640891 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:17:47.640893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:17:47.640963 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:17:47.641198 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:17:47.641201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:17:47.641203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:17:47.641380 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:17:47.641484 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:47.641697 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 04:17:47.641805 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 04:17:47.647753 21071 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:17:47.648959 21071 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:17:47.657610 21071 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e48276b247114054a0c33162fb19fee4: Generated new TSK 0
I20260924 04:17:47.658403 21071 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:17:47.663515 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 20993
2026-09-24T04:17:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1292 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790223216265354-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:17:47 dist-test-slave-p401 krb5kdc[21092](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:17:47 dist-test-slave-p401 krb5kdc[21092](info): set up 2 sockets
Sep 24 04:17:47 dist-test-slave-p401 krb5kdc[21092](info): commencing operation
krb5kdc: starting...
W20260924 04:17:48.611254 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790223216265354-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790223216265354-392-0/krb5kdc/test-user.keytab.
Sep 24 04:17:48 dist-test-slave-p401 krb5kdc[21092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:17:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:17:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790223216265354-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790223216265354-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:17:48.722998 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790223216265354-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790223216265354-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790223216265354-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45301
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790223216265354-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45301
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:17:48.835361 21105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:17:48.835638 21105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:17:48.835680 21105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:17:48.839725 21105 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:17:48.839792 21105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:17:48.839809 21105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:17:48.839824 21105 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:17:48.839838 21105 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:17:48.846033 21105 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790223216265354-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790223216265354-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45301
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790223216265354-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790223216265354-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:17:48.847054 21105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:17:48.848099 21105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:17:48.856349 21113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:17:48.856423 21111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:17:48.856349 21110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:17:48.856525 21105 server_base.cc:1034] running on GCE node
I20260924 04:17:48.857075 21105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:17:48.858096 21105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:17:48.859285 21105 hybrid_clock.cc:648] HybridClock initialized: now 1790223468859259 us; error 35 us; skew 500 ppm
Sep 24 04:17:48 dist-test-slave-p401 krb5kdc[21092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223468, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:17:48.868430 21105 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:17:48.869340 21105 webserver.cc:460] Webserver started at http://127.0.0.1:41227/ using document root <none> and password file <none>
I20260924 04:17:48.869865 21105 fs_manager.cc:359] Metadata directory not provided
I20260924 04:17:48.869925 21105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:17:48.870118 21105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:17:48.872171 21105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790223216265354-392-0/minicluster-data/master-0/data/instance:
uuid: "e57b1ea85fe94418aded81f8bdd8435e"
format_stamp: "Formatted at 2026-09-24 04:17:48 on dist-test-slave-p401"
server_key: "6815195f6ad543700fc9be93bc975b75"
server_key_iv: "57311163a7cace5d948af765c7a86d11"
server_key_version: "encryptionkey@0"
I20260924 04:17:48.872684 21105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790223216265354-392-0/minicluster-data/master-0/wal/instance:
uuid: "e57b1ea85fe94418aded81f8bdd8435e"
format_stamp: "Formatted at 2026-09-24 04:17:48 on dist-test-slave-p401"
server_key: "6815195f6ad543700fc9be93bc975b75"
server_key_iv: "57311163a7cace5d948af765c7a86d11"
server_key_version: "encryptionkey@0"
I20260924 04:17:48.876263 21105 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:17:48.878592 21120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:48.879910 21105 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:17:48.880029 21105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790223216265354-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790223216265354-392-0/minicluster-data/master-0/wal
uuid: "e57b1ea85fe94418aded81f8bdd8435e"
format_stamp: "Formatted at 2026-09-24 04:17:48 on dist-test-slave-p401"
server_key: "6815195f6ad543700fc9be93bc975b75"
server_key_iv: "57311163a7cace5d948af765c7a86d11"
server_key_version: "encryptionkey@0"
I20260924 04:17:48.880120 21105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790223216265354-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790223216265354-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790223216265354-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:17:48.898838 21105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:17:48.901813 21105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:17:48.901983 21105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:17:48.908628 21105 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45301
I20260924 04:17:48.908653 21172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45301 every 8 connection(s)
I20260924 04:17:48.909534 21105 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:17:48.912179 21173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:17:48.914006 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 21105
I20260924 04:17:48.914139 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790223216265354-392-0/minicluster-data/master-0/wal/instance
I20260924 04:17:48.914376 392 external_mini_cluster.cc:1428] Setting key 423f337540ff695a25e394b996bd715f
I20260924 04:17:48.918284 21173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e57b1ea85fe94418aded81f8bdd8435e: Bootstrap starting.
I20260924 04:17:48.920759 21173 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e57b1ea85fe94418aded81f8bdd8435e: Neither blocks nor log segments found. Creating new log.
I20260924 04:17:48.921545 21173 log.cc:824] T 00000000000000000000000000000000 P e57b1ea85fe94418aded81f8bdd8435e: Log is configured to *not* fsync() on all Append() calls
I20260924 04:17:48.922981 21176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:48.915623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35565 (local address 127.0.0.1:45301)
0924 04:17:48.916000 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:48.916016 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:48.916051 (+ 35us) server_negotiation.cc:407] Connection header received
0924 04:17:48.916675 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:48.916691 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:48.917015 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:48.917294 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:17:48.917644 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:48.918598 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:48.919771 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:48.920223 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:48.920332 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:48.921131 (+ 799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:17:48.921137 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:17:48.921152 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:17:48.921624 (+ 472us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:17:48.921689 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:17:48.921719 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:17:48.921811 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:48.921915 (+ 104us) server_negotiation.cc:299] Negotiation successful
0924 04:17:48.922112 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":97,"threads_started":1}
I20260924 04:17:48.923575 21173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e57b1ea85fe94418aded81f8bdd8435e: No bootstrap required, opened a new log
I20260924 04:17:48.925666 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:17:48.927292 21173 raft_consensus.cc:348] T 00000000000000000000000000000000 P e57b1ea85fe94418aded81f8bdd8435e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e57b1ea85fe94418aded81f8bdd8435e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45301 } }
I20260924 04:17:48.927598 21173 raft_consensus.cc:374] T 00000000000000000000000000000000 P e57b1ea85fe94418aded81f8bdd8435e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:17:48.927656 21173 raft_consensus.cc:729] T 00000000000000000000000000000000 P e57b1ea85fe94418aded81f8bdd8435e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e57b1ea85fe94418aded81f8bdd8435e, State: Initialized, Role: FOLLOWER
I20260924 04:17:48.928121 21173 consensus_queue.cc:260] T 00000000000000000000000000000000 P e57b1ea85fe94418aded81f8bdd8435e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e57b1ea85fe94418aded81f8bdd8435e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45301 } }
I20260924 04:17:48.928279 21173 raft_consensus.cc:388] T 00000000000000000000000000000000 P e57b1ea85fe94418aded81f8bdd8435e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:17:48.928344 21173 raft_consensus.cc:482] T 00000000000000000000000000000000 P e57b1ea85fe94418aded81f8bdd8435e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:17:48.928433 21173 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e57b1ea85fe94418aded81f8bdd8435e [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:17:48.929435 21173 raft_consensus.cc:504] T 00000000000000000000000000000000 P e57b1ea85fe94418aded81f8bdd8435e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e57b1ea85fe94418aded81f8bdd8435e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45301 } }
I20260924 04:17:48.929781 21173 leader_election.cc:302] T 00000000000000000000000000000000 P e57b1ea85fe94418aded81f8bdd8435e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e57b1ea85fe94418aded81f8bdd8435e; no voters:
I20260924 04:17:48.930076 21173 leader_election.cc:288] T 00000000000000000000000000000000 P e57b1ea85fe94418aded81f8bdd8435e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:17:48.930150 21183 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e57b1ea85fe94418aded81f8bdd8435e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:17:48.930380 21183 raft_consensus.cc:686] T 00000000000000000000000000000000 P e57b1ea85fe94418aded81f8bdd8435e [term 1 LEADER]: Becoming Leader. State: Replica: e57b1ea85fe94418aded81f8bdd8435e, State: Running, Role: LEADER
I20260924 04:17:48.930709 21183 consensus_queue.cc:237] T 00000000000000000000000000000000 P e57b1ea85fe94418aded81f8bdd8435e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e57b1ea85fe94418aded81f8bdd8435e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45301 } }
I20260924 04:17:48.931361 21176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:48.927437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52315 (local address 127.0.0.1:45301)
0924 04:17:48.927542 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:48.927546 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:48.927688 (+ 142us) server_negotiation.cc:407] Connection header received
0924 04:17:48.927786 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:48.927789 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:48.927828 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:48.927894 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:17:48.928152 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:48.928765 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:48.929781 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:48.930050 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:48.930138 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:48.930916 (+ 778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:17:48.930920 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:17:48.930923 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:17:48.930936 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:17:48.930963 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:17:48.930975 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:17:48.931014 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:48.931165 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 04:17:48.931212 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 04:17:48.931401 21173 sys_catalog.cc:564] T 00000000000000000000000000000000 P e57b1ea85fe94418aded81f8bdd8435e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:17:48.932583 21185 sys_catalog.cc:455] T 00000000000000000000000000000000 P e57b1ea85fe94418aded81f8bdd8435e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e57b1ea85fe94418aded81f8bdd8435e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e57b1ea85fe94418aded81f8bdd8435e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45301 } } }
I20260924 04:17:48.932713 21185 sys_catalog.cc:458] T 00000000000000000000000000000000 P e57b1ea85fe94418aded81f8bdd8435e [sys.catalog]: This master's current role is: LEADER
I20260924 04:17:48.932890 21184 sys_catalog.cc:455] T 00000000000000000000000000000000 P e57b1ea85fe94418aded81f8bdd8435e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e57b1ea85fe94418aded81f8bdd8435e. Latest consensus state: current_term: 1 leader_uuid: "e57b1ea85fe94418aded81f8bdd8435e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e57b1ea85fe94418aded81f8bdd8435e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45301 } } }
I20260924 04:17:48.933162 21184 sys_catalog.cc:458] T 00000000000000000000000000000000 P e57b1ea85fe94418aded81f8bdd8435e [sys.catalog]: This master's current role is: LEADER
I20260924 04:17:48.933302 21197 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:17:48.936615 21197 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:17:48.941653 21197 catalog_manager.cc:1347] Generated new cluster ID: ae915d22d00e4ed58df5dae60d0eaa40
I20260924 04:17:48.941712 21197 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:17:48.989974 21197 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:17:48.990942 21197 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:17:48.995008 21197 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e57b1ea85fe94418aded81f8bdd8435e: Generated new TSK 0
I20260924 04:17:48.995688 21197 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:17:49.000010 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 21105
2026-09-24T04:17:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1336 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790223216265354-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:17:49 dist-test-slave-p401 krb5kdc[21204](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:17:49 dist-test-slave-p401 krb5kdc[21204](info): set up 2 sockets
Sep 24 04:17:49 dist-test-slave-p401 krb5kdc[21204](info): commencing operation
krb5kdc: starting...
W20260924 04:17:49.943331 392 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790223216265354-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790223216265354-392-0/krb5kdc/test-user.keytab.
Sep 24 04:17:50 dist-test-slave-p401 krb5kdc[21204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:17:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:17:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790223216265354-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790223216265354-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:17:50.059815 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790223216265354-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790223216265354-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790223216265354-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40847
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790223216265354-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:17:50.164448 21217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:17:50.164717 21217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:17:50.164758 21217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:17:50.168812 21217 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:17:50.168892 21217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:17:50.168911 21217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:17:50.168927 21217 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:17:50.168941 21217 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:17:50.174463 21217 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790223216265354-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790223216265354-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790223216265354-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790223216265354-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:17:50.175583 21217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:17:50.176610 21217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:17:50.184763 21222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:17:50.184804 21223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:17:50.184891 21217 server_base.cc:1034] running on GCE node
W20260924 04:17:50.184772 21225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:17:50.185451 21217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:17:50.186481 21217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:17:50.187631 21217 hybrid_clock.cc:648] HybridClock initialized: now 1790223470187614 us; error 29 us; skew 500 ppm
Sep 24 04:17:50 dist-test-slave-p401 krb5kdc[21204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:17:50.190402 21217 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:17:50.191306 21217 webserver.cc:460] Webserver started at http://127.0.0.1:39611/ using document root <none> and password file <none>
I20260924 04:17:50.191855 21217 fs_manager.cc:359] Metadata directory not provided
I20260924 04:17:50.191923 21217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:17:50.192147 21217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:17:50.194139 21217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790223216265354-392-0/minicluster-data/master-0/data/instance:
uuid: "eb9391b047ba432194298ad8650bf96e"
format_stamp: "Formatted at 2026-09-24 04:17:50 on dist-test-slave-p401"
server_key: "6fa8073cdf85bfec42654219fae5ea6b"
server_key_iv: "10a37c553913cc2ebefdd15335b9ee70"
server_key_version: "encryptionkey@0"
I20260924 04:17:50.194658 21217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790223216265354-392-0/minicluster-data/master-0/wal/instance:
uuid: "eb9391b047ba432194298ad8650bf96e"
format_stamp: "Formatted at 2026-09-24 04:17:50 on dist-test-slave-p401"
server_key: "6fa8073cdf85bfec42654219fae5ea6b"
server_key_iv: "10a37c553913cc2ebefdd15335b9ee70"
server_key_version: "encryptionkey@0"
I20260924 04:17:50.198223 21217 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 04:17:50.200815 21232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:50.202191 21217 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 04:17:50.202347 21217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790223216265354-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790223216265354-392-0/minicluster-data/master-0/wal
uuid: "eb9391b047ba432194298ad8650bf96e"
format_stamp: "Formatted at 2026-09-24 04:17:50 on dist-test-slave-p401"
server_key: "6fa8073cdf85bfec42654219fae5ea6b"
server_key_iv: "10a37c553913cc2ebefdd15335b9ee70"
server_key_version: "encryptionkey@0"
I20260924 04:17:50.202457 21217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790223216265354-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790223216265354-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790223216265354-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:17:50.206967 21217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:17:50.209973 21217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:17:50.210141 21217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:17:50.216691 21217 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40847
I20260924 04:17:50.216706 21284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40847 every 8 connection(s)
I20260924 04:17:50.217619 21217 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:17:50.220480 21285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:17:50.225333 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 21217
I20260924 04:17:50.225449 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790223216265354-392-0/minicluster-data/master-0/wal/instance
I20260924 04:17:50.225701 392 external_mini_cluster.cc:1428] Setting key 45822d16f5af95c6684f6833d0cfc041
I20260924 04:17:50.226220 21285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb9391b047ba432194298ad8650bf96e: Bootstrap starting.
I20260924 04:17:50.228861 21285 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb9391b047ba432194298ad8650bf96e: Neither blocks nor log segments found. Creating new log.
I20260924 04:17:50.229758 21285 log.cc:824] T 00000000000000000000000000000000 P eb9391b047ba432194298ad8650bf96e: Log is configured to *not* fsync() on all Append() calls
I20260924 04:17:50.231330 21288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:50.227094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34829 (local address 127.0.0.1:40847)
0924 04:17:50.227559 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:50.227576 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:50.227628 (+ 52us) server_negotiation.cc:407] Connection header received
0924 04:17:50.228437 (+ 809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:50.228462 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:50.228838 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:50.229222 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:17:50.229888 (+ 666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:17:50.229894 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:17:50.229908 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:17:50.229918 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:17:50.229979 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:17:50.230006 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:17:50.230114 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:50.230265 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 04:17:50.230407 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":127,"threads_started":1}
I20260924 04:17:50.232002 21285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb9391b047ba432194298ad8650bf96e: No bootstrap required, opened a new log
I20260924 04:17:50.234061 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:17:50.235764 21285 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb9391b047ba432194298ad8650bf96e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9391b047ba432194298ad8650bf96e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40847 } }
I20260924 04:17:50.236018 21285 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb9391b047ba432194298ad8650bf96e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:17:50.236128 21285 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb9391b047ba432194298ad8650bf96e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb9391b047ba432194298ad8650bf96e, State: Initialized, Role: FOLLOWER
I20260924 04:17:50.236572 21285 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb9391b047ba432194298ad8650bf96e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9391b047ba432194298ad8650bf96e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40847 } }
I20260924 04:17:50.236708 21285 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb9391b047ba432194298ad8650bf96e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:17:50.236774 21285 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb9391b047ba432194298ad8650bf96e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:17:50.236884 21285 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb9391b047ba432194298ad8650bf96e [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:17:50.237599 21288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:50.236027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59801 (local address 127.0.0.1:40847)
0924 04:17:50.236154 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:50.236159 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:50.236346 (+ 187us) server_negotiation.cc:407] Connection header received
0924 04:17:50.236447 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:50.236450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:50.236482 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:50.236548 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:17:50.237254 (+ 706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:17:50.237258 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:17:50.237260 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:17:50.237271 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:17:50.237298 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:17:50.237308 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:17:50.237378 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:50.237448 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 04:17:50.237485 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 04:17:50.237885 21285 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb9391b047ba432194298ad8650bf96e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9391b047ba432194298ad8650bf96e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40847 } }
I20260924 04:17:50.238241 21285 leader_election.cc:302] T 00000000000000000000000000000000 P eb9391b047ba432194298ad8650bf96e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb9391b047ba432194298ad8650bf96e; no voters:
I20260924 04:17:50.238533 21285 leader_election.cc:288] T 00000000000000000000000000000000 P eb9391b047ba432194298ad8650bf96e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:17:50.238646 21295 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb9391b047ba432194298ad8650bf96e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:17:50.238832 21295 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb9391b047ba432194298ad8650bf96e [term 1 LEADER]: Becoming Leader. State: Replica: eb9391b047ba432194298ad8650bf96e, State: Running, Role: LEADER
I20260924 04:17:50.239152 21295 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb9391b047ba432194298ad8650bf96e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9391b047ba432194298ad8650bf96e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40847 } }
I20260924 04:17:50.239742 21285 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb9391b047ba432194298ad8650bf96e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:17:50.240772 21297 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb9391b047ba432194298ad8650bf96e [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb9391b047ba432194298ad8650bf96e. Latest consensus state: current_term: 1 leader_uuid: "eb9391b047ba432194298ad8650bf96e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9391b047ba432194298ad8650bf96e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40847 } } }
I20260924 04:17:50.240891 21297 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb9391b047ba432194298ad8650bf96e [sys.catalog]: This master's current role is: LEADER
I20260924 04:17:50.241367 21304 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:17:50.241669 21296 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb9391b047ba432194298ad8650bf96e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb9391b047ba432194298ad8650bf96e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9391b047ba432194298ad8650bf96e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40847 } } }
I20260924 04:17:50.241772 21296 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb9391b047ba432194298ad8650bf96e [sys.catalog]: This master's current role is: LEADER
I20260924 04:17:50.244048 21304 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:17:50.249519 21304 catalog_manager.cc:1347] Generated new cluster ID: 2cf1d8f715b04d49aa462c38f3c0a6b5
I20260924 04:17:50.249593 21304 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:17:50.273149 21304 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:17:50.273510 21304 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:17:50.284972 21304 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb9391b047ba432194298ad8650bf96e: Generated new TSK 0
I20260924 04:17:50.285820 21304 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:17:50.290072 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 21217
2026-09-24T04:17:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1289 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790223216265354-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:17:50 dist-test-slave-p401 krb5kdc[21316](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:17:50 dist-test-slave-p401 krb5kdc[21316](info): set up 2 sockets
Sep 24 04:17:50 dist-test-slave-p401 krb5kdc[21316](info): commencing operation
krb5kdc: starting...
W20260924 04:17:51.239223 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790223216265354-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790223216265354-392-0/krb5kdc/test-user.keytab.
Sep 24 04:17:51 dist-test-slave-p401 krb5kdc[21316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:17:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:17:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790223216265354-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790223216265354-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:17:51.353796 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790223216265354-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790223216265354-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790223216265354-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35213
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790223216265354-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35213
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:17:51.459446 21329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:17:51.459708 21329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:17:51.459751 21329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:17:51.463743 21329 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:17:51.463814 21329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:17:51.463833 21329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:17:51.463848 21329 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:17:51.463862 21329 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:17:51.469205 21329 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790223216265354-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790223216265354-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790223216265354-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790223216265354-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:17:51.470271 21329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:17:51.471302 21329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:17:51.479580 21329 server_base.cc:1034] running on GCE node
W20260924 04:17:51.479537 21337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:17:51.479508 21334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:17:51.479523 21335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:17:51.480268 21329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:17:51.481261 21329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:17:51.482405 21329 hybrid_clock.cc:648] HybridClock initialized: now 1790223471482373 us; error 46 us; skew 500 ppm
Sep 24 04:17:51 dist-test-slave-p401 krb5kdc[21316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:17:51.485000 21329 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:17:51.485829 21329 webserver.cc:460] Webserver started at http://127.0.0.1:41131/ using document root <none> and password file <none>
I20260924 04:17:51.486330 21329 fs_manager.cc:359] Metadata directory not provided
I20260924 04:17:51.486392 21329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:17:51.486567 21329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:17:51.488523 21329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790223216265354-392-0/minicluster-data/master-0/data/instance:
uuid: "6ee6972dc0d84f18a1366048585de4b3"
format_stamp: "Formatted at 2026-09-24 04:17:51 on dist-test-slave-p401"
server_key: "7b0da9ba635065298dfb4a0add7e5398"
server_key_iv: "b4d963ba865d2ebc017b71788a729ea3"
server_key_version: "encryptionkey@0"
I20260924 04:17:51.489033 21329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790223216265354-392-0/minicluster-data/master-0/wal/instance:
uuid: "6ee6972dc0d84f18a1366048585de4b3"
format_stamp: "Formatted at 2026-09-24 04:17:51 on dist-test-slave-p401"
server_key: "7b0da9ba635065298dfb4a0add7e5398"
server_key_iv: "b4d963ba865d2ebc017b71788a729ea3"
server_key_version: "encryptionkey@0"
I20260924 04:17:51.492412 21329 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:17:51.494518 21344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:51.495661 21329 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:17:51.495764 21329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790223216265354-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790223216265354-392-0/minicluster-data/master-0/wal
uuid: "6ee6972dc0d84f18a1366048585de4b3"
format_stamp: "Formatted at 2026-09-24 04:17:51 on dist-test-slave-p401"
server_key: "7b0da9ba635065298dfb4a0add7e5398"
server_key_iv: "b4d963ba865d2ebc017b71788a729ea3"
server_key_version: "encryptionkey@0"
I20260924 04:17:51.495867 21329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790223216265354-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790223216265354-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790223216265354-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:17:51.508271 21329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:17:51.510993 21329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:17:51.511154 21329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:17:51.517776 21329 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35213
I20260924 04:17:51.517831 21396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35213 every 8 connection(s)
I20260924 04:17:51.518651 21329 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:17:51.519349 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 21329
I20260924 04:17:51.519469 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790223216265354-392-0/minicluster-data/master-0/wal/instance
I20260924 04:17:51.519721 392 external_mini_cluster.cc:1428] Setting key 51278390497a4f03a7d16020f75479b2
I20260924 04:17:51.521642 21397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:17:51.527621 21397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ee6972dc0d84f18a1366048585de4b3: Bootstrap starting.
Sep 24 04:17:51 dist-test-slave-p401 krb5kdc[21316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 04:17:51.529771 21397 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ee6972dc0d84f18a1366048585de4b3: Neither blocks nor log segments found. Creating new log.
I20260924 04:17:51.530416 21397 log.cc:824] T 00000000000000000000000000000000 P 6ee6972dc0d84f18a1366048585de4b3: Log is configured to *not* fsync() on all Append() calls
I20260924 04:17:51.532398 21397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ee6972dc0d84f18a1366048585de4b3: No bootstrap required, opened a new log
I20260924 04:17:51.533584 21400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:51.521025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:48579 (local address 127.0.0.1:35213)
0924 04:17:51.521463 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:51.521478 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:51.521691 (+ 213us) server_negotiation.cc:407] Connection header received
0924 04:17:51.522419 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:51.522435 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:51.522757 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:51.523025 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:17:51.523873 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:51.524856 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:51.525963 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:51.526409 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:51.526513 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:51.529169 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:17:51.529174 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:17:51.529186 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:17:51.529211 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:17:51.530734 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:17:51.531262 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:17:51.531268 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:17:51.531273 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:17:51.531387 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:17:51.531677 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:17:51.531680 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:17:51.531681 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:17:51.532039 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:17:51.532192 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:51.532490 (+ 298us) server_negotiation.cc:299] Negotiation successful
0924 04:17:51.532716 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":139,"threads_started":1}
I20260924 04:17:51.535266 21397 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ee6972dc0d84f18a1366048585de4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ee6972dc0d84f18a1366048585de4b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35213 } }
I20260924 04:17:51.535513 21397 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ee6972dc0d84f18a1366048585de4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:17:51.535583 21397 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ee6972dc0d84f18a1366048585de4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ee6972dc0d84f18a1366048585de4b3, State: Initialized, Role: FOLLOWER
I20260924 04:17:51.535991 21397 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ee6972dc0d84f18a1366048585de4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ee6972dc0d84f18a1366048585de4b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35213 } }
I20260924 04:17:51.536144 21397 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ee6972dc0d84f18a1366048585de4b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:17:51.536310 21397 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ee6972dc0d84f18a1366048585de4b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:17:51.536361 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:17:51.536417 21397 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ee6972dc0d84f18a1366048585de4b3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:17:51.537375 21397 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ee6972dc0d84f18a1366048585de4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ee6972dc0d84f18a1366048585de4b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35213 } }
I20260924 04:17:51.537755 21397 leader_election.cc:302] T 00000000000000000000000000000000 P 6ee6972dc0d84f18a1366048585de4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ee6972dc0d84f18a1366048585de4b3; no voters:
I20260924 04:17:51.538043 21397 leader_election.cc:288] T 00000000000000000000000000000000 P 6ee6972dc0d84f18a1366048585de4b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:17:51.538146 21406 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ee6972dc0d84f18a1366048585de4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:17:51.538370 21406 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ee6972dc0d84f18a1366048585de4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 6ee6972dc0d84f18a1366048585de4b3, State: Running, Role: LEADER
I20260924 04:17:51.538718 21406 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ee6972dc0d84f18a1366048585de4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ee6972dc0d84f18a1366048585de4b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35213 } }
I20260924 04:17:51.539394 21397 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ee6972dc0d84f18a1366048585de4b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:17:51.540319 21409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ee6972dc0d84f18a1366048585de4b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ee6972dc0d84f18a1366048585de4b3. Latest consensus state: current_term: 1 leader_uuid: "6ee6972dc0d84f18a1366048585de4b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ee6972dc0d84f18a1366048585de4b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35213 } } }
I20260924 04:17:51.540678 21409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ee6972dc0d84f18a1366048585de4b3 [sys.catalog]: This master's current role is: LEADER
I20260924 04:17:51.540584 21408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ee6972dc0d84f18a1366048585de4b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ee6972dc0d84f18a1366048585de4b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ee6972dc0d84f18a1366048585de4b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35213 } } }
I20260924 04:17:51.540982 21408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ee6972dc0d84f18a1366048585de4b3 [sys.catalog]: This master's current role is: LEADER
W20260924 04:17:51.544598 21423 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6ee6972dc0d84f18a1366048585de4b3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:17:51.544683 21423 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:17:51.544965 21416 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:17:51.545922 21416 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:17:51.549335 21400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:51.538191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:45965 (local address 127.0.0.1:35213)
0924 04:17:51.538291 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:51.538295 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:51.540120 (+ 1825us) server_negotiation.cc:407] Connection header received
0924 04:17:51.540329 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:51.540333 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:51.540387 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:51.540443 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:17:51.541402 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:51.542143 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:51.543296 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:51.543640 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:51.543722 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:51.545669 (+ 1947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:17:51.545673 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:17:51.545676 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:17:51.545702 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:17:51.547434 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:17:51.548026 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:17:51.548030 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:17:51.548032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:17:51.548123 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:17:51.548457 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:17:51.548461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:17:51.548463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:17:51.548661 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:17:51.548805 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:51.549078 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 04:17:51.549173 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 04:17:51.552208 21416 catalog_manager.cc:1347] Generated new cluster ID: bb16893682b14a0889aba7619a28cfec
I20260924 04:17:51.552271 21416 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:17:51.569613 21416 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:17:51.570869 21416 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:17:51.577445 21416 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ee6972dc0d84f18a1366048585de4b3: Generated new TSK 0
I20260924 04:17:51.578191 21416 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:17:51.587147 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 21329
W20260924 04:17:51.591738 21314 connection.cc:535] client connection to 127.0.0.1:35213 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35213): Connection reset by peer (error 104)
2026-09-24T04:17:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1297 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790223216265354-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:17:51 dist-test-slave-p401 krb5kdc[21428](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:17:51 dist-test-slave-p401 krb5kdc[21428](info): set up 2 sockets
Sep 24 04:17:51 dist-test-slave-p401 krb5kdc[21428](info): commencing operation
krb5kdc: starting...
W20260924 04:17:52.527082 392 mini_kdc.cc:121] Time spent starting KDC: real 0.922s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790223216265354-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790223216265354-392-0/krb5kdc/test-user.keytab.
Sep 24 04:17:52 dist-test-slave-p401 krb5kdc[21428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:17:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:17:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790223216265354-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790223216265354-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:17:52.640336 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790223216265354-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790223216265354-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790223216265354-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45601
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790223216265354-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45601
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:17:52.747727 21441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:17:52.747982 21441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:17:52.748039 21441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:17:52.752012 21441 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:17:52.752079 21441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:17:52.752102 21441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:17:52.752126 21441 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:17:52.752154 21441 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:17:52.757494 21441 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790223216265354-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790223216265354-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45601
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790223216265354-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790223216265354-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:17:52.758546 21441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:17:52.759588 21441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:17:52.767621 21447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:17:52.767699 21449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:17:52.767742 21441 server_base.cc:1034] running on GCE node
W20260924 04:17:52.767621 21446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:17:52.768478 21441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:17:52.769584 21441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:17:52.770818 21441 hybrid_clock.cc:648] HybridClock initialized: now 1790223472770799 us; error 32 us; skew 500 ppm
Sep 24 04:17:52 dist-test-slave-p401 krb5kdc[21428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:17:52.773546 21441 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:17:52.774385 21441 webserver.cc:460] Webserver started at http://127.0.0.1:46365/ using document root <none> and password file <none>
I20260924 04:17:52.774897 21441 fs_manager.cc:359] Metadata directory not provided
I20260924 04:17:52.774956 21441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:17:52.775128 21441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:17:52.777190 21441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790223216265354-392-0/minicluster-data/master-0/data/instance:
uuid: "454a610ca1c04b3a8e27c83a4da87f28"
format_stamp: "Formatted at 2026-09-24 04:17:52 on dist-test-slave-p401"
server_key: "a2938792b7444be8644931fd8d5602fc"
server_key_iv: "6a725a18c5abeec7e174c422cc3eb44f"
server_key_version: "encryptionkey@0"
I20260924 04:17:52.777699 21441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790223216265354-392-0/minicluster-data/master-0/wal/instance:
uuid: "454a610ca1c04b3a8e27c83a4da87f28"
format_stamp: "Formatted at 2026-09-24 04:17:52 on dist-test-slave-p401"
server_key: "a2938792b7444be8644931fd8d5602fc"
server_key_iv: "6a725a18c5abeec7e174c422cc3eb44f"
server_key_version: "encryptionkey@0"
I20260924 04:17:52.781286 21441 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:17:52.783754 21456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:52.785082 21441 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:17:52.785211 21441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790223216265354-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790223216265354-392-0/minicluster-data/master-0/wal
uuid: "454a610ca1c04b3a8e27c83a4da87f28"
format_stamp: "Formatted at 2026-09-24 04:17:52 on dist-test-slave-p401"
server_key: "a2938792b7444be8644931fd8d5602fc"
server_key_iv: "6a725a18c5abeec7e174c422cc3eb44f"
server_key_version: "encryptionkey@0"
I20260924 04:17:52.785326 21441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790223216265354-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790223216265354-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790223216265354-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:17:52.822307 21441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:17:52.825443 21441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:17:52.825613 21441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:17:52.832090 21441 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45601
I20260924 04:17:52.832108 21508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45601 every 8 connection(s)
I20260924 04:17:52.833029 21441 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:17:52.835656 21509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:17:52.835987 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 21441
I20260924 04:17:52.836100 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790223216265354-392-0/minicluster-data/master-0/wal/instance
I20260924 04:17:52.836385 392 external_mini_cluster.cc:1428] Setting key 88b9adb89d6e61c24e631bd7a77c28d6
I20260924 04:17:52.841797 21509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 454a610ca1c04b3a8e27c83a4da87f28: Bootstrap starting.
I20260924 04:17:52.843901 21509 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 454a610ca1c04b3a8e27c83a4da87f28: Neither blocks nor log segments found. Creating new log.
I20260924 04:17:52.844581 21509 log.cc:824] T 00000000000000000000000000000000 P 454a610ca1c04b3a8e27c83a4da87f28: Log is configured to *not* fsync() on all Append() calls
I20260924 04:17:52.844812 21512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:52.837669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:34917 (local address 127.0.0.1:45601)
0924 04:17:52.838089 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:52.838103 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:52.838137 (+ 34us) server_negotiation.cc:407] Connection header received
0924 04:17:52.838794 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:52.838810 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:52.839134 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:52.839516 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:17:52.839911 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:52.840861 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:52.841719 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:52.842200 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:52.842336 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:52.843155 (+ 819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:17:52.843160 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:17:52.843172 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:17:52.843582 (+ 410us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:17:52.843634 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:17:52.843657 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:17:52.843774 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:52.843962 (+ 188us) server_negotiation.cc:299] Negotiation successful
0924 04:17:52.844095 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":142,"threads_started":1}
I20260924 04:17:52.846472 21509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 454a610ca1c04b3a8e27c83a4da87f28: No bootstrap required, opened a new log
I20260924 04:17:52.847512 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:17:52.850034 21509 raft_consensus.cc:348] T 00000000000000000000000000000000 P 454a610ca1c04b3a8e27c83a4da87f28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "454a610ca1c04b3a8e27c83a4da87f28" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45601 } }
I20260924 04:17:52.850234 21509 raft_consensus.cc:374] T 00000000000000000000000000000000 P 454a610ca1c04b3a8e27c83a4da87f28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:17:52.850281 21509 raft_consensus.cc:729] T 00000000000000000000000000000000 P 454a610ca1c04b3a8e27c83a4da87f28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 454a610ca1c04b3a8e27c83a4da87f28, State: Initialized, Role: FOLLOWER
I20260924 04:17:52.850679 21509 consensus_queue.cc:260] T 00000000000000000000000000000000 P 454a610ca1c04b3a8e27c83a4da87f28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "454a610ca1c04b3a8e27c83a4da87f28" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45601 } }
I20260924 04:17:52.850805 21509 raft_consensus.cc:388] T 00000000000000000000000000000000 P 454a610ca1c04b3a8e27c83a4da87f28 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:17:52.850869 21509 raft_consensus.cc:482] T 00000000000000000000000000000000 P 454a610ca1c04b3a8e27c83a4da87f28 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:17:52.850966 21509 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 454a610ca1c04b3a8e27c83a4da87f28 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:17:52.852030 21509 raft_consensus.cc:504] T 00000000000000000000000000000000 P 454a610ca1c04b3a8e27c83a4da87f28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "454a610ca1c04b3a8e27c83a4da87f28" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45601 } }
I20260924 04:17:52.852460 21509 leader_election.cc:302] T 00000000000000000000000000000000 P 454a610ca1c04b3a8e27c83a4da87f28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 454a610ca1c04b3a8e27c83a4da87f28; no voters:
I20260924 04:17:52.852787 21509 leader_election.cc:288] T 00000000000000000000000000000000 P 454a610ca1c04b3a8e27c83a4da87f28 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:17:52.852864 21519 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 454a610ca1c04b3a8e27c83a4da87f28 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:17:52.853030 21519 raft_consensus.cc:686] T 00000000000000000000000000000000 P 454a610ca1c04b3a8e27c83a4da87f28 [term 1 LEADER]: Becoming Leader. State: Replica: 454a610ca1c04b3a8e27c83a4da87f28, State: Running, Role: LEADER
I20260924 04:17:52.853313 21512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:52.849360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:48427 (local address 127.0.0.1:45601)
0924 04:17:52.849457 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:52.849461 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:52.849617 (+ 156us) server_negotiation.cc:407] Connection header received
0924 04:17:52.849722 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:52.849724 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:52.849763 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:52.849823 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:17:52.850054 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:52.850631 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:52.851638 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:52.852002 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:52.852093 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:52.852861 (+ 768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:17:52.852865 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:17:52.852868 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:17:52.852880 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:17:52.852906 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:17:52.852919 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:17:52.852993 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:52.853147 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 04:17:52.853190 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 04:17:52.853456 21519 consensus_queue.cc:237] T 00000000000000000000000000000000 P 454a610ca1c04b3a8e27c83a4da87f28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "454a610ca1c04b3a8e27c83a4da87f28" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45601 } }
I20260924 04:17:52.854045 21509 sys_catalog.cc:564] T 00000000000000000000000000000000 P 454a610ca1c04b3a8e27c83a4da87f28 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:17:52.857161 21520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 454a610ca1c04b3a8e27c83a4da87f28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "454a610ca1c04b3a8e27c83a4da87f28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "454a610ca1c04b3a8e27c83a4da87f28" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45601 } } }
I20260924 04:17:52.857161 21521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 454a610ca1c04b3a8e27c83a4da87f28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 454a610ca1c04b3a8e27c83a4da87f28. Latest consensus state: current_term: 1 leader_uuid: "454a610ca1c04b3a8e27c83a4da87f28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "454a610ca1c04b3a8e27c83a4da87f28" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45601 } } }
I20260924 04:17:52.857295 21520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 454a610ca1c04b3a8e27c83a4da87f28 [sys.catalog]: This master's current role is: LEADER
I20260924 04:17:52.857300 21521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 454a610ca1c04b3a8e27c83a4da87f28 [sys.catalog]: This master's current role is: LEADER
W20260924 04:17:52.858589 21533 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 454a610ca1c04b3a8e27c83a4da87f28: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:17:52.858654 21533 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:17:52.858728 21536 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:17:52.859476 21536 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:17:52.864178 21536 catalog_manager.cc:1347] Generated new cluster ID: 6e7206901c1344cba236deef4caf61db
I20260924 04:17:52.864238 21536 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:17:52.873579 21536 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:17:52.874717 21536 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:17:52.880873 21536 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 454a610ca1c04b3a8e27c83a4da87f28: Generated new TSK 0
I20260924 04:17:52.881620 21536 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:17:52.885141 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 21441
W20260924 04:17:52.889662 21426 connection.cc:535] client connection to 127.0.0.1:45601 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45601): Connection reset by peer (error 104)
2026-09-24T04:17:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1297 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790223216265354-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:17:52 dist-test-slave-p401 krb5kdc[21540](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:17:52 dist-test-slave-p401 krb5kdc[21540](info): set up 2 sockets
Sep 24 04:17:52 dist-test-slave-p401 krb5kdc[21540](info): commencing operation
krb5kdc: starting...
W20260924 04:17:53.843266 392 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790223216265354-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790223216265354-392-0/krb5kdc/test-user.keytab.
Sep 24 04:17:53 dist-test-slave-p401 krb5kdc[21540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:17:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:17:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790223216265354-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790223216265354-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:17:53.956646 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790223216265354-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790223216265354-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790223216265354-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43683
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790223216265354-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43683
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:17:54.061425 21553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:17:54.061687 21553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:17:54.061743 21553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:17:54.065704 21553 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:17:54.065760 21553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:17:54.065786 21553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:17:54.065806 21553 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:17:54.065840 21553 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:17:54.070967 21553 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790223216265354-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790223216265354-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43683
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790223216265354-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790223216265354-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:17:54.072007 21553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:17:54.072983 21553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:17:54.080932 21559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:17:54.080955 21558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:17:54.080951 21561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:17:54.081236 21553 server_base.cc:1034] running on GCE node
I20260924 04:17:54.081820 21553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:17:54.082868 21553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:17:54.084048 21553 hybrid_clock.cc:648] HybridClock initialized: now 1790223474084026 us; error 30 us; skew 500 ppm
Sep 24 04:17:54 dist-test-slave-p401 krb5kdc[21540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:17:54.091995 21553 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:17:54.092902 21553 webserver.cc:460] Webserver started at http://127.0.0.1:39165/ using document root <none> and password file <none>
I20260924 04:17:54.093429 21553 fs_manager.cc:359] Metadata directory not provided
I20260924 04:17:54.093488 21553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:17:54.093667 21553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:17:54.095666 21553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790223216265354-392-0/minicluster-data/master-0/data/instance:
uuid: "a5afd993043948408b6b175a8bd07e0a"
format_stamp: "Formatted at 2026-09-24 04:17:54 on dist-test-slave-p401"
server_key: "cf2244abb819bd03502f21b85283739c"
server_key_iv: "5e5a746797a117eb3ac3960b98c8837f"
server_key_version: "encryptionkey@0"
I20260924 04:17:54.096155 21553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790223216265354-392-0/minicluster-data/master-0/wal/instance:
uuid: "a5afd993043948408b6b175a8bd07e0a"
format_stamp: "Formatted at 2026-09-24 04:17:54 on dist-test-slave-p401"
server_key: "cf2244abb819bd03502f21b85283739c"
server_key_iv: "5e5a746797a117eb3ac3960b98c8837f"
server_key_version: "encryptionkey@0"
I20260924 04:17:54.099681 21553 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 04:17:54.101958 21568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:54.103128 21553 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:17:54.103222 21553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790223216265354-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790223216265354-392-0/minicluster-data/master-0/wal
uuid: "a5afd993043948408b6b175a8bd07e0a"
format_stamp: "Formatted at 2026-09-24 04:17:54 on dist-test-slave-p401"
server_key: "cf2244abb819bd03502f21b85283739c"
server_key_iv: "5e5a746797a117eb3ac3960b98c8837f"
server_key_version: "encryptionkey@0"
I20260924 04:17:54.103346 21553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790223216265354-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790223216265354-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790223216265354-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:17:54.107642 21553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:17:54.110397 21553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:17:54.110535 21553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:17:54.117167 21553 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43683
I20260924 04:17:54.117210 21620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43683 every 8 connection(s)
I20260924 04:17:54.118108 21553 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:17:54.120766 21621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:17:54.122188 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 21553
I20260924 04:17:54.122313 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790223216265354-392-0/minicluster-data/master-0/wal/instance
I20260924 04:17:54.122586 392 external_mini_cluster.cc:1428] Setting key e5086e81923397297a050b9278a959b6
I20260924 04:17:54.127053 21621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5afd993043948408b6b175a8bd07e0a: Bootstrap starting.
I20260924 04:17:54.127940 21624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:54.123976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:50087 (local address 127.0.0.1:43683)
0924 04:17:54.124381 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:54.124400 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:54.124441 (+ 41us) server_negotiation.cc:407] Connection header received
0924 04:17:54.125207 (+ 766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:54.125230 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:54.125582 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:54.125952 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:17:54.126593 (+ 641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:17:54.126599 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:17:54.126613 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:17:54.126624 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:17:54.126682 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:17:54.126708 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:17:54.126844 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:54.126911 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 04:17:54.127062 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":109,"threads_started":1}
I20260924 04:17:54.129506 21621 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5afd993043948408b6b175a8bd07e0a: Neither blocks nor log segments found. Creating new log.
I20260924 04:17:54.130290 21621 log.cc:824] T 00000000000000000000000000000000 P a5afd993043948408b6b175a8bd07e0a: Log is configured to *not* fsync() on all Append() calls
I20260924 04:17:54.130633 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:17:54.132351 21621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5afd993043948408b6b175a8bd07e0a: No bootstrap required, opened a new log
I20260924 04:17:54.133937 21624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:54.132797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:48719 (local address 127.0.0.1:43683)
0924 04:17:54.132905 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:54.132908 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:54.132919 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:17:54.132950 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:54.132952 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:54.132976 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:54.133033 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:17:54.133636 (+ 603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:17:54.133639 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:17:54.133641 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:17:54.133651 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:17:54.133673 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:17:54.133683 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:17:54.133739 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:54.133819 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 04:17:54.133848 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 04:17:54.136265 21621 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5afd993043948408b6b175a8bd07e0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5afd993043948408b6b175a8bd07e0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43683 } }
I20260924 04:17:54.136531 21621 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5afd993043948408b6b175a8bd07e0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:17:54.136590 21621 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5afd993043948408b6b175a8bd07e0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5afd993043948408b6b175a8bd07e0a, State: Initialized, Role: FOLLOWER
I20260924 04:17:54.136960 21621 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5afd993043948408b6b175a8bd07e0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5afd993043948408b6b175a8bd07e0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43683 } }
I20260924 04:17:54.137076 21621 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5afd993043948408b6b175a8bd07e0a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:17:54.137136 21621 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5afd993043948408b6b175a8bd07e0a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:17:54.137226 21621 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5afd993043948408b6b175a8bd07e0a [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:17:54.138012 21621 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5afd993043948408b6b175a8bd07e0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5afd993043948408b6b175a8bd07e0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43683 } }
I20260924 04:17:54.138309 21621 leader_election.cc:302] T 00000000000000000000000000000000 P a5afd993043948408b6b175a8bd07e0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5afd993043948408b6b175a8bd07e0a; no voters:
I20260924 04:17:54.138576 21621 leader_election.cc:288] T 00000000000000000000000000000000 P a5afd993043948408b6b175a8bd07e0a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:17:54.138662 21631 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5afd993043948408b6b175a8bd07e0a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:17:54.138849 21631 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5afd993043948408b6b175a8bd07e0a [term 1 LEADER]: Becoming Leader. State: Replica: a5afd993043948408b6b175a8bd07e0a, State: Running, Role: LEADER
I20260924 04:17:54.139174 21631 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5afd993043948408b6b175a8bd07e0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5afd993043948408b6b175a8bd07e0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43683 } }
I20260924 04:17:54.139642 21621 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5afd993043948408b6b175a8bd07e0a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:17:54.140648 21632 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5afd993043948408b6b175a8bd07e0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5afd993043948408b6b175a8bd07e0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5afd993043948408b6b175a8bd07e0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43683 } } }
I20260924 04:17:54.140767 21632 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5afd993043948408b6b175a8bd07e0a [sys.catalog]: This master's current role is: LEADER
I20260924 04:17:54.141137 21640 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:17:54.141685 21633 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5afd993043948408b6b175a8bd07e0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5afd993043948408b6b175a8bd07e0a. Latest consensus state: current_term: 1 leader_uuid: "a5afd993043948408b6b175a8bd07e0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5afd993043948408b6b175a8bd07e0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43683 } } }
I20260924 04:17:54.141778 21633 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5afd993043948408b6b175a8bd07e0a [sys.catalog]: This master's current role is: LEADER
I20260924 04:17:54.143772 21640 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:17:54.148979 21640 catalog_manager.cc:1347] Generated new cluster ID: 0a159688f4c545009b2f10a963dc500a
I20260924 04:17:54.149091 21640 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:17:54.170495 21640 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:17:54.170823 21640 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:17:54.177937 21640 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5afd993043948408b6b175a8bd07e0a: Generated new TSK 0
I20260924 04:17:54.178577 21640 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:17:54.185434 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 21553
2026-09-24T04:17:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1300 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790223216265354-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:17:54 dist-test-slave-p401 krb5kdc[21652](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:17:54 dist-test-slave-p401 krb5kdc[21652](info): set up 2 sockets
Sep 24 04:17:54 dist-test-slave-p401 krb5kdc[21652](info): commencing operation
krb5kdc: starting...
W20260924 04:17:55.147164 392 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790223216265354-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790223216265354-392-0/krb5kdc/test-user.keytab.
Sep 24 04:17:55 dist-test-slave-p401 krb5kdc[21652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:17:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:17:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790223216265354-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790223216265354-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:17:55.282251 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790223216265354-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790223216265354-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790223216265354-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34645
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790223216265354-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34645
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:17:55.387665 21665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:17:55.387940 21665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:17:55.388002 21665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:17:55.392066 21665 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:17:55.392138 21665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:17:55.392164 21665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:17:55.392189 21665 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:17:55.392217 21665 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:17:55.397617 21665 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790223216265354-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790223216265354-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34645
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790223216265354-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790223216265354-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:17:55.398661 21665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:17:55.399708 21665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:17:55.408396 21673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:17:55.408401 21671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:17:55.408457 21665 server_base.cc:1034] running on GCE node
W20260924 04:17:55.408401 21670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:17:55.409245 21665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:17:55.410392 21665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:17:55.411556 21665 hybrid_clock.cc:648] HybridClock initialized: now 1790223475411532 us; error 38 us; skew 500 ppm
Sep 24 04:17:55 dist-test-slave-p401 krb5kdc[21652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223475, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:17:55.414279 21665 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:17:55.415141 21665 webserver.cc:460] Webserver started at http://127.0.0.1:42349/ using document root <none> and password file <none>
I20260924 04:17:55.415704 21665 fs_manager.cc:359] Metadata directory not provided
I20260924 04:17:55.415773 21665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:17:55.415977 21665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:17:55.418035 21665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790223216265354-392-0/minicluster-data/master-0/data/instance:
uuid: "3babb20cb7454dbcb3ca46e2f5fb24a7"
format_stamp: "Formatted at 2026-09-24 04:17:55 on dist-test-slave-p401"
server_key: "3bef071320f378b00f247365d26e3edd"
server_key_iv: "5110b695d1f8f78f855244d475047682"
server_key_version: "encryptionkey@0"
I20260924 04:17:55.418581 21665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790223216265354-392-0/minicluster-data/master-0/wal/instance:
uuid: "3babb20cb7454dbcb3ca46e2f5fb24a7"
format_stamp: "Formatted at 2026-09-24 04:17:55 on dist-test-slave-p401"
server_key: "3bef071320f378b00f247365d26e3edd"
server_key_iv: "5110b695d1f8f78f855244d475047682"
server_key_version: "encryptionkey@0"
I20260924 04:17:55.422199 21665 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 04:17:55.424582 21680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:55.425799 21665 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:17:55.425909 21665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790223216265354-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790223216265354-392-0/minicluster-data/master-0/wal
uuid: "3babb20cb7454dbcb3ca46e2f5fb24a7"
format_stamp: "Formatted at 2026-09-24 04:17:55 on dist-test-slave-p401"
server_key: "3bef071320f378b00f247365d26e3edd"
server_key_iv: "5110b695d1f8f78f855244d475047682"
server_key_version: "encryptionkey@0"
I20260924 04:17:55.426000 21665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790223216265354-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790223216265354-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790223216265354-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:17:55.444474 21665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:17:55.447476 21665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:17:55.447645 21665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:17:55.454165 21665 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34645
I20260924 04:17:55.454185 21732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34645 every 8 connection(s)
I20260924 04:17:55.455150 21665 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:17:55.457878 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 21665
I20260924 04:17:55.457913 21733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:17:55.458025 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790223216265354-392-0/minicluster-data/master-0/wal/instance
I20260924 04:17:55.458307 392 external_mini_cluster.cc:1428] Setting key 11c52d390ad9529a250e594ff84414f7
I20260924 04:17:55.464063 21733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3babb20cb7454dbcb3ca46e2f5fb24a7: Bootstrap starting.
I20260924 04:17:55.466172 21733 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3babb20cb7454dbcb3ca46e2f5fb24a7: Neither blocks nor log segments found. Creating new log.
Sep 24 04:17:55 dist-test-slave-p401 krb5kdc[21652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 04:17:55.466868 21733 log.cc:824] T 00000000000000000000000000000000 P 3babb20cb7454dbcb3ca46e2f5fb24a7: Log is configured to *not* fsync() on all Append() calls
I20260924 04:17:55.468667 21733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3babb20cb7454dbcb3ca46e2f5fb24a7: No bootstrap required, opened a new log
I20260924 04:17:55.471777 21736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:55.459718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:50009 (local address 127.0.0.1:34645)
0924 04:17:55.460154 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:55.460168 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:55.460204 (+ 36us) server_negotiation.cc:407] Connection header received
0924 04:17:55.461035 (+ 831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:55.461053 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:55.461429 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:55.461739 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:17:55.462535 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:55.463541 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:55.464392 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:55.464850 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:55.464996 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:55.467373 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:17:55.467377 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:17:55.467390 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:17:55.467416 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:17:55.469013 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:17:55.469486 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:17:55.469491 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:17:55.469496 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:17:55.469603 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:17:55.469913 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:17:55.469923 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:17:55.469924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:17:55.470307 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:17:55.470493 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:55.470795 (+ 302us) server_negotiation.cc:299] Negotiation successful
0924 04:17:55.470998 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":141,"threads_started":1}
I20260924 04:17:55.472208 21733 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3babb20cb7454dbcb3ca46e2f5fb24a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3babb20cb7454dbcb3ca46e2f5fb24a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34645 } }
I20260924 04:17:55.472429 21733 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3babb20cb7454dbcb3ca46e2f5fb24a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:17:55.472503 21733 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3babb20cb7454dbcb3ca46e2f5fb24a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3babb20cb7454dbcb3ca46e2f5fb24a7, State: Initialized, Role: FOLLOWER
I20260924 04:17:55.472950 21733 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3babb20cb7454dbcb3ca46e2f5fb24a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3babb20cb7454dbcb3ca46e2f5fb24a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34645 } }
I20260924 04:17:55.473105 21733 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3babb20cb7454dbcb3ca46e2f5fb24a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:17:55.473163 21733 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3babb20cb7454dbcb3ca46e2f5fb24a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:17:55.473253 21733 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3babb20cb7454dbcb3ca46e2f5fb24a7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:17:55.474262 21733 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3babb20cb7454dbcb3ca46e2f5fb24a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3babb20cb7454dbcb3ca46e2f5fb24a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34645 } }
I20260924 04:17:55.474601 21733 leader_election.cc:302] T 00000000000000000000000000000000 P 3babb20cb7454dbcb3ca46e2f5fb24a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3babb20cb7454dbcb3ca46e2f5fb24a7; no voters:
I20260924 04:17:55.474771 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:17:55.474927 21733 leader_election.cc:288] T 00000000000000000000000000000000 P 3babb20cb7454dbcb3ca46e2f5fb24a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:17:55.474993 21738 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3babb20cb7454dbcb3ca46e2f5fb24a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:17:55.475158 21738 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3babb20cb7454dbcb3ca46e2f5fb24a7 [term 1 LEADER]: Becoming Leader. State: Replica: 3babb20cb7454dbcb3ca46e2f5fb24a7, State: Running, Role: LEADER
I20260924 04:17:55.475467 21738 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3babb20cb7454dbcb3ca46e2f5fb24a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3babb20cb7454dbcb3ca46e2f5fb24a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34645 } }
I20260924 04:17:55.476177 21733 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3babb20cb7454dbcb3ca46e2f5fb24a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:17:55.477039 21739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3babb20cb7454dbcb3ca46e2f5fb24a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3babb20cb7454dbcb3ca46e2f5fb24a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3babb20cb7454dbcb3ca46e2f5fb24a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34645 } } }
I20260924 04:17:55.477175 21739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3babb20cb7454dbcb3ca46e2f5fb24a7 [sys.catalog]: This master's current role is: LEADER
I20260924 04:17:55.477480 21740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3babb20cb7454dbcb3ca46e2f5fb24a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3babb20cb7454dbcb3ca46e2f5fb24a7. Latest consensus state: current_term: 1 leader_uuid: "3babb20cb7454dbcb3ca46e2f5fb24a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3babb20cb7454dbcb3ca46e2f5fb24a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34645 } } }
I20260924 04:17:55.477527 21751 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:17:55.477566 21740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3babb20cb7454dbcb3ca46e2f5fb24a7 [sys.catalog]: This master's current role is: LEADER
I20260924 04:17:55.480592 21751 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:17:55.486487 21751 catalog_manager.cc:1347] Generated new cluster ID: 1742ad56c83a4b1896d3eb682c6105ec
I20260924 04:17:55.486543 21751 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:17:55.487614 21736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:55.476632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:48255 (local address 127.0.0.1:34645)
0924 04:17:55.476748 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:55.476752 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:55.478668 (+ 1916us) server_negotiation.cc:407] Connection header received
0924 04:17:55.478774 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:55.478777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:55.478822 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:55.478898 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:17:55.479712 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:55.480438 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:55.481614 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:55.481934 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:55.482033 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:55.484078 (+ 2045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:17:55.484081 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:17:55.484083 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:17:55.484104 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:17:55.485776 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:17:55.486235 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:17:55.486238 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:17:55.486240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:17:55.486324 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:17:55.486672 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:17:55.486675 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:17:55.486677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:17:55.486909 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:17:55.487056 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:55.487366 (+ 310us) server_negotiation.cc:299] Negotiation successful
0924 04:17:55.487464 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 04:17:55.493101 21751 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:17:55.493902 21751 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:17:55.499616 21751 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3babb20cb7454dbcb3ca46e2f5fb24a7: Generated new TSK 0
I20260924 04:17:55.500279 21751 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:17:55.509945 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 21665
W20260924 04:17:55.514520 21650 connection.cc:535] client connection to 127.0.0.1:34645 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34645): Connection reset by peer (error 104)
2026-09-24T04:17:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1324 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790223216265354-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:17:55 dist-test-slave-p401 krb5kdc[21764](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:17:55 dist-test-slave-p401 krb5kdc[21764](info): set up 2 sockets
Sep 24 04:17:55 dist-test-slave-p401 krb5kdc[21764](info): commencing operation
krb5kdc: starting...
W20260924 04:17:56.459276 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790223216265354-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790223216265354-392-0/krb5kdc/test-user.keytab.
Sep 24 04:17:56 dist-test-slave-p401 krb5kdc[21764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:17:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:17:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790223216265354-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790223216265354-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:17:56.576100 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790223216265354-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790223216265354-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790223216265354-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36877
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790223216265354-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:17:56.682526 21777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:17:56.682777 21777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:17:56.682818 21777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:17:56.686693 21777 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:17:56.686751 21777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:17:56.686770 21777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:17:56.686784 21777 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:17:56.686798 21777 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:17:56.691903 21777 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790223216265354-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790223216265354-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36877
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790223216265354-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790223216265354-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:17:56.692834 21777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:17:56.693751 21777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:17:56.701562 21782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:17:56.701638 21777 server_base.cc:1034] running on GCE node
W20260924 04:17:56.701565 21783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:17:56.701663 21785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:17:56.702299 21777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:17:56.703354 21777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:17:56.704504 21777 hybrid_clock.cc:648] HybridClock initialized: now 1790223476704480 us; error 37 us; skew 500 ppm
Sep 24 04:17:56 dist-test-slave-p401 krb5kdc[21764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223476, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:17:56.706944 21777 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:17:56.707789 21777 webserver.cc:460] Webserver started at http://127.0.0.1:46683/ using document root <none> and password file <none>
I20260924 04:17:56.708279 21777 fs_manager.cc:359] Metadata directory not provided
I20260924 04:17:56.708335 21777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:17:56.708490 21777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:17:56.710422 21777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790223216265354-392-0/minicluster-data/master-0/data/instance:
uuid: "ec18cd7f1181427fb07f29900feffd75"
format_stamp: "Formatted at 2026-09-24 04:17:56 on dist-test-slave-p401"
server_key: "51023936aa49e7b25ee9e92ba8952a05"
server_key_iv: "4e19d766f44afe326c3bf07ba7eb7649"
server_key_version: "encryptionkey@0"
I20260924 04:17:56.710915 21777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790223216265354-392-0/minicluster-data/master-0/wal/instance:
uuid: "ec18cd7f1181427fb07f29900feffd75"
format_stamp: "Formatted at 2026-09-24 04:17:56 on dist-test-slave-p401"
server_key: "51023936aa49e7b25ee9e92ba8952a05"
server_key_iv: "4e19d766f44afe326c3bf07ba7eb7649"
server_key_version: "encryptionkey@0"
I20260924 04:17:56.714391 21777 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 04:17:56.716756 21792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:56.717916 21777 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:17:56.718020 21777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790223216265354-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790223216265354-392-0/minicluster-data/master-0/wal
uuid: "ec18cd7f1181427fb07f29900feffd75"
format_stamp: "Formatted at 2026-09-24 04:17:56 on dist-test-slave-p401"
server_key: "51023936aa49e7b25ee9e92ba8952a05"
server_key_iv: "4e19d766f44afe326c3bf07ba7eb7649"
server_key_version: "encryptionkey@0"
I20260924 04:17:56.718116 21777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790223216265354-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790223216265354-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790223216265354-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:17:56.737758 21777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:17:56.740720 21777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:17:56.740902 21777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:17:56.747450 21777 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36877
I20260924 04:17:56.747476 21844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36877 every 8 connection(s)
I20260924 04:17:56.748358 21777 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:17:56.750979 21845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:17:56.752404 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 21777
I20260924 04:17:56.752528 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790223216265354-392-0/minicluster-data/master-0/wal/instance
I20260924 04:17:56.752771 392 external_mini_cluster.cc:1428] Setting key 7b28131c8063cd9874c3c30182bf002f
I20260924 04:17:56.757298 21845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec18cd7f1181427fb07f29900feffd75: Bootstrap starting.
I20260924 04:17:56.759773 21845 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec18cd7f1181427fb07f29900feffd75: Neither blocks nor log segments found. Creating new log.
I20260924 04:17:56.760859 21845 log.cc:824] T 00000000000000000000000000000000 P ec18cd7f1181427fb07f29900feffd75: Log is configured to *not* fsync() on all Append() calls
I20260924 04:17:56.762210 21848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:56.754056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:59033 (local address 127.0.0.1:36877)
0924 04:17:56.754455 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:56.754474 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:56.754516 (+ 42us) server_negotiation.cc:407] Connection header received
0924 04:17:56.755347 (+ 831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:56.755369 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:56.755810 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:56.756192 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:17:56.756585 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:56.757986 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:56.758812 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:56.759470 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:56.759616 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:56.760319 (+ 703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:17:56.760325 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:17:56.760340 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:17:56.760820 (+ 480us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:17:56.760887 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:17:56.760918 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:17:56.761043 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:56.761251 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 04:17:56.761364 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":108,"threads_started":1}
I20260924 04:17:56.762996 21845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec18cd7f1181427fb07f29900feffd75: No bootstrap required, opened a new log
I20260924 04:17:56.765014 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:17:56.766635 21845 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec18cd7f1181427fb07f29900feffd75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec18cd7f1181427fb07f29900feffd75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36877 } }
I20260924 04:17:56.766934 21845 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec18cd7f1181427fb07f29900feffd75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:17:56.766995 21845 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec18cd7f1181427fb07f29900feffd75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec18cd7f1181427fb07f29900feffd75, State: Initialized, Role: FOLLOWER
I20260924 04:17:56.767445 21845 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec18cd7f1181427fb07f29900feffd75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec18cd7f1181427fb07f29900feffd75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36877 } }
I20260924 04:17:56.767575 21845 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec18cd7f1181427fb07f29900feffd75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:17:56.767669 21845 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec18cd7f1181427fb07f29900feffd75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:17:56.767776 21845 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec18cd7f1181427fb07f29900feffd75 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:17:56.768777 21845 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec18cd7f1181427fb07f29900feffd75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec18cd7f1181427fb07f29900feffd75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36877 } }
I20260924 04:17:56.769084 21845 leader_election.cc:302] T 00000000000000000000000000000000 P ec18cd7f1181427fb07f29900feffd75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec18cd7f1181427fb07f29900feffd75; no voters:
I20260924 04:17:56.769394 21845 leader_election.cc:288] T 00000000000000000000000000000000 P ec18cd7f1181427fb07f29900feffd75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:17:56.769482 21855 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec18cd7f1181427fb07f29900feffd75 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:17:56.769682 21855 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec18cd7f1181427fb07f29900feffd75 [term 1 LEADER]: Becoming Leader. State: Replica: ec18cd7f1181427fb07f29900feffd75, State: Running, Role: LEADER
I20260924 04:17:56.770007 21855 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec18cd7f1181427fb07f29900feffd75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec18cd7f1181427fb07f29900feffd75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36877 } }
I20260924 04:17:56.770709 21845 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec18cd7f1181427fb07f29900feffd75 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:17:56.771080 21848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:56.766823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:40941 (local address 127.0.0.1:36877)
0924 04:17:56.766921 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:56.766925 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:56.767138 (+ 213us) server_negotiation.cc:407] Connection header received
0924 04:17:56.767276 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:56.767279 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:56.767320 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:56.767386 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:17:56.767642 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:56.768349 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:56.769155 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:56.769509 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:56.769608 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:56.770575 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:17:56.770578 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:17:56.770581 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:17:56.770593 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:17:56.770619 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:17:56.770632 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:17:56.770684 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:56.770922 (+ 238us) server_negotiation.cc:299] Negotiation successful
0924 04:17:56.770955 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 04:17:56.771668 21856 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec18cd7f1181427fb07f29900feffd75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec18cd7f1181427fb07f29900feffd75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec18cd7f1181427fb07f29900feffd75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36877 } } }
I20260924 04:17:56.771802 21856 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec18cd7f1181427fb07f29900feffd75 [sys.catalog]: This master's current role is: LEADER
I20260924 04:17:56.772644 21857 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec18cd7f1181427fb07f29900feffd75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec18cd7f1181427fb07f29900feffd75. Latest consensus state: current_term: 1 leader_uuid: "ec18cd7f1181427fb07f29900feffd75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec18cd7f1181427fb07f29900feffd75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36877 } } }
I20260924 04:17:56.773415 21857 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec18cd7f1181427fb07f29900feffd75 [sys.catalog]: This master's current role is: LEADER
W20260924 04:17:56.775005 21870 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ec18cd7f1181427fb07f29900feffd75: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:17:56.775077 21870 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:17:56.775287 21864 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:17:56.776211 21864 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:17:56.781559 21864 catalog_manager.cc:1347] Generated new cluster ID: 576c4b60f75c469396ab96be83e8bf59
I20260924 04:17:56.781627 21864 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:17:56.795919 21864 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:17:56.797062 21864 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:17:56.801357 21864 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec18cd7f1181427fb07f29900feffd75: Generated new TSK 0
I20260924 04:17:56.801975 21864 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:17:56.809085 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 21777
W20260924 04:17:56.813720 21762 connection.cc:535] client connection to 127.0.0.1:36877 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36877): Connection reset by peer (error 104)
2026-09-24T04:17:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1299 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790223216265354-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:17:56 dist-test-slave-p401 krb5kdc[21876](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:17:56 dist-test-slave-p401 krb5kdc[21876](info): set up 2 sockets
Sep 24 04:17:56 dist-test-slave-p401 krb5kdc[21876](info): commencing operation
krb5kdc: starting...
W20260924 04:17:57.759212 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790223216265354-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790223216265354-392-0/krb5kdc/test-user.keytab.
Sep 24 04:17:57 dist-test-slave-p401 krb5kdc[21876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:17:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:17:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790223216265354-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790223216265354-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:17:57.873447 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790223216265354-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790223216265354-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790223216265354-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32917
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790223216265354-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32917
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:17:57.980350 21889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:17:57.980625 21889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:17:57.980666 21889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:17:57.984651 21889 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:17:57.984709 21889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:17:57.984727 21889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:17:57.984743 21889 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:17:57.984771 21889 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:17:57.990001 21889 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790223216265354-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790223216265354-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32917
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790223216265354-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790223216265354-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:17:57.991011 21889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:17:57.992020 21889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:17:57.999927 21895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:17:57.999922 21894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:17:57.999960 21897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:17:58.000279 21889 server_base.cc:1034] running on GCE node
I20260924 04:17:58.000872 21889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:17:58.001915 21889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:17:58.003077 21889 hybrid_clock.cc:648] HybridClock initialized: now 1790223478003059 us; error 31 us; skew 500 ppm
Sep 24 04:17:58 dist-test-slave-p401 krb5kdc[21876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223478, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:17:58.005673 21889 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:17:58.006556 21889 webserver.cc:460] Webserver started at http://127.0.0.1:44279/ using document root <none> and password file <none>
I20260924 04:17:58.007073 21889 fs_manager.cc:359] Metadata directory not provided
I20260924 04:17:58.007136 21889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:17:58.007371 21889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:17:58.009375 21889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790223216265354-392-0/minicluster-data/master-0/data/instance:
uuid: "824ce1ff1913407abecc684d89e435e6"
format_stamp: "Formatted at 2026-09-24 04:17:58 on dist-test-slave-p401"
server_key: "1f5fab5f44854ae7c50be9d50a8fe32a"
server_key_iv: "4a39948b23b2ddc8038b47b456dc6091"
server_key_version: "encryptionkey@0"
I20260924 04:17:58.009919 21889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790223216265354-392-0/minicluster-data/master-0/wal/instance:
uuid: "824ce1ff1913407abecc684d89e435e6"
format_stamp: "Formatted at 2026-09-24 04:17:58 on dist-test-slave-p401"
server_key: "1f5fab5f44854ae7c50be9d50a8fe32a"
server_key_iv: "4a39948b23b2ddc8038b47b456dc6091"
server_key_version: "encryptionkey@0"
I20260924 04:17:58.013612 21889 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:17:58.016000 21904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:58.017154 21889 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:17:58.017269 21889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790223216265354-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790223216265354-392-0/minicluster-data/master-0/wal
uuid: "824ce1ff1913407abecc684d89e435e6"
format_stamp: "Formatted at 2026-09-24 04:17:58 on dist-test-slave-p401"
server_key: "1f5fab5f44854ae7c50be9d50a8fe32a"
server_key_iv: "4a39948b23b2ddc8038b47b456dc6091"
server_key_version: "encryptionkey@0"
I20260924 04:17:58.017393 21889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790223216265354-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790223216265354-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790223216265354-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:17:58.021845 21889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:17:58.024890 21889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:17:58.025053 21889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:17:58.031719 21889 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32917
I20260924 04:17:58.031744 21956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32917 every 8 connection(s)
I20260924 04:17:58.032630 21889 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:17:58.035290 21957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:17:58.039798 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 21889
I20260924 04:17:58.039942 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790223216265354-392-0/minicluster-data/master-0/wal/instance
I20260924 04:17:58.040246 392 external_mini_cluster.cc:1428] Setting key 357581756eaf60cdef21c3ff20a5c900
I20260924 04:17:58.040858 21957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 824ce1ff1913407abecc684d89e435e6: Bootstrap starting.
I20260924 04:17:58.043140 21957 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 824ce1ff1913407abecc684d89e435e6: Neither blocks nor log segments found. Creating new log.
I20260924 04:17:58.043938 21957 log.cc:824] T 00000000000000000000000000000000 P 824ce1ff1913407abecc684d89e435e6: Log is configured to *not* fsync() on all Append() calls
I20260924 04:17:58.045146 21960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:58.041612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:58037 (local address 127.0.0.1:32917)
0924 04:17:58.042030 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:58.042043 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:58.042077 (+ 34us) server_negotiation.cc:407] Connection header received
0924 04:17:58.042677 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:58.042692 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:58.042980 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:58.043296 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:17:58.044011 (+ 715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:17:58.044016 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:17:58.044026 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:17:58.044036 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:17:58.044104 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:17:58.044130 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:17:58.044220 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:58.044334 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 04:17:58.044488 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":126,"threads_started":1}
I20260924 04:17:58.045794 21957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 824ce1ff1913407abecc684d89e435e6: No bootstrap required, opened a new log
I20260924 04:17:58.047737 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:17:58.049398 21957 raft_consensus.cc:348] T 00000000000000000000000000000000 P 824ce1ff1913407abecc684d89e435e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824ce1ff1913407abecc684d89e435e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32917 } }
I20260924 04:17:58.049680 21957 raft_consensus.cc:374] T 00000000000000000000000000000000 P 824ce1ff1913407abecc684d89e435e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:17:58.049752 21957 raft_consensus.cc:729] T 00000000000000000000000000000000 P 824ce1ff1913407abecc684d89e435e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 824ce1ff1913407abecc684d89e435e6, State: Initialized, Role: FOLLOWER
I20260924 04:17:58.050184 21957 consensus_queue.cc:260] T 00000000000000000000000000000000 P 824ce1ff1913407abecc684d89e435e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824ce1ff1913407abecc684d89e435e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32917 } }
I20260924 04:17:58.050333 21957 raft_consensus.cc:388] T 00000000000000000000000000000000 P 824ce1ff1913407abecc684d89e435e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:17:58.050398 21957 raft_consensus.cc:482] T 00000000000000000000000000000000 P 824ce1ff1913407abecc684d89e435e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:17:58.050499 21957 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 824ce1ff1913407abecc684d89e435e6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:17:58.051007 21960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:58.049588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:51967 (local address 127.0.0.1:32917)
0924 04:17:58.049763 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:58.049767 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:58.049797 (+ 30us) server_negotiation.cc:407] Connection header received
0924 04:17:58.049918 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:58.049921 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:58.049951 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:58.050011 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:17:58.050663 (+ 652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:17:58.050666 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:17:58.050669 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:17:58.050680 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:17:58.050703 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:17:58.050714 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:17:58.050781 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:58.050861 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 04:17:58.050899 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260924 04:17:58.051539 21957 raft_consensus.cc:504] T 00000000000000000000000000000000 P 824ce1ff1913407abecc684d89e435e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824ce1ff1913407abecc684d89e435e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32917 } }
I20260924 04:17:58.051888 21957 leader_election.cc:302] T 00000000000000000000000000000000 P 824ce1ff1913407abecc684d89e435e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 824ce1ff1913407abecc684d89e435e6; no voters:
I20260924 04:17:58.052142 21957 leader_election.cc:288] T 00000000000000000000000000000000 P 824ce1ff1913407abecc684d89e435e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:17:58.052229 21967 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 824ce1ff1913407abecc684d89e435e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:17:58.052425 21967 raft_consensus.cc:686] T 00000000000000000000000000000000 P 824ce1ff1913407abecc684d89e435e6 [term 1 LEADER]: Becoming Leader. State: Replica: 824ce1ff1913407abecc684d89e435e6, State: Running, Role: LEADER
I20260924 04:17:58.052762 21967 consensus_queue.cc:237] T 00000000000000000000000000000000 P 824ce1ff1913407abecc684d89e435e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824ce1ff1913407abecc684d89e435e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32917 } }
I20260924 04:17:58.053251 21957 sys_catalog.cc:564] T 00000000000000000000000000000000 P 824ce1ff1913407abecc684d89e435e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:17:58.054328 21969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 824ce1ff1913407abecc684d89e435e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 824ce1ff1913407abecc684d89e435e6. Latest consensus state: current_term: 1 leader_uuid: "824ce1ff1913407abecc684d89e435e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824ce1ff1913407abecc684d89e435e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32917 } } }
I20260924 04:17:58.054459 21969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 824ce1ff1913407abecc684d89e435e6 [sys.catalog]: This master's current role is: LEADER
I20260924 04:17:58.054327 21968 sys_catalog.cc:455] T 00000000000000000000000000000000 P 824ce1ff1913407abecc684d89e435e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "824ce1ff1913407abecc684d89e435e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824ce1ff1913407abecc684d89e435e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32917 } } }
I20260924 04:17:58.054716 21968 sys_catalog.cc:458] T 00000000000000000000000000000000 P 824ce1ff1913407abecc684d89e435e6 [sys.catalog]: This master's current role is: LEADER
I20260924 04:17:58.054800 21976 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:17:58.057684 21976 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:17:58.062668 21976 catalog_manager.cc:1347] Generated new cluster ID: dc35fe0197a5489998dfda1c1b46a7b2
I20260924 04:17:58.062742 21976 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:17:58.089226 21976 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:17:58.089589 21976 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:17:58.094380 21976 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 824ce1ff1913407abecc684d89e435e6: Generated new TSK 0
I20260924 04:17:58.095068 21976 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:17:58.106627 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 21889
2026-09-24T04:17:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1297 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790223216265354-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:17:58 dist-test-slave-p401 krb5kdc[21988](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:17:58 dist-test-slave-p401 krb5kdc[21988](info): set up 2 sockets
Sep 24 04:17:58 dist-test-slave-p401 krb5kdc[21988](info): commencing operation
krb5kdc: starting...
W20260924 04:17:59.055141 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790223216265354-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790223216265354-392-0/krb5kdc/test-user.keytab.
Sep 24 04:17:59 dist-test-slave-p401 krb5kdc[21988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:17:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:17:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:17:59.169569 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790223216265354-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790223216265354-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790223216265354-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34419
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:17:59.275849 22001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:17:59.276115 22001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:17:59.276157 22001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:17:59.280129 22001 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:17:59.280193 22001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:17:59.280211 22001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:17:59.280234 22001 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:17:59.280252 22001 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:17:59.285425 22001 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790223216265354-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790223216265354-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34419
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790223216265354-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:17:59.286415 22001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:17:59.287401 22001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:17:59.295434 22006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:17:59.295490 22001 server_base.cc:1034] running on GCE node
W20260924 04:17:59.295444 22007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:17:59.295413 22009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:17:59.296123 22001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:17:59.297170 22001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:17:59.298348 22001 hybrid_clock.cc:648] HybridClock initialized: now 1790223479298329 us; error 30 us; skew 500 ppm
Sep 24 04:17:59 dist-test-slave-p401 krb5kdc[21988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223479, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:17:59.308516 22001 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:17:59.309445 22001 webserver.cc:460] Webserver started at http://127.0.98.62:39581/ using document root <none> and password file <none>
I20260924 04:17:59.309971 22001 fs_manager.cc:359] Metadata directory not provided
I20260924 04:17:59.310036 22001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:17:59.310230 22001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:17:59.312286 22001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790223216265354-392-0/minicluster-data/master-0/data/instance:
uuid: "09c6a03e501d4f02a32588655fe2a09c"
format_stamp: "Formatted at 2026-09-24 04:17:59 on dist-test-slave-p401"
server_key: "af6a126e23d18dc5cb718f88f1fca6fb"
server_key_iv: "8e7f397800ecabec987b4f28842316b9"
server_key_version: "encryptionkey@0"
I20260924 04:17:59.312794 22001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790223216265354-392-0/minicluster-data/master-0/wal/instance:
uuid: "09c6a03e501d4f02a32588655fe2a09c"
format_stamp: "Formatted at 2026-09-24 04:17:59 on dist-test-slave-p401"
server_key: "af6a126e23d18dc5cb718f88f1fca6fb"
server_key_iv: "8e7f397800ecabec987b4f28842316b9"
server_key_version: "encryptionkey@0"
I20260924 04:17:59.316329 22001 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260924 04:17:59.318759 22016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:17:59.319931 22001 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 04:17:59.320041 22001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790223216265354-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790223216265354-392-0/minicluster-data/master-0/wal
uuid: "09c6a03e501d4f02a32588655fe2a09c"
format_stamp: "Formatted at 2026-09-24 04:17:59 on dist-test-slave-p401"
server_key: "af6a126e23d18dc5cb718f88f1fca6fb"
server_key_iv: "8e7f397800ecabec987b4f28842316b9"
server_key_version: "encryptionkey@0"
I20260924 04:17:59.320130 22001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790223216265354-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790223216265354-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790223216265354-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:17:59.333576 22001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:17:59.336199 22001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:17:59.336372 22001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:17:59.342916 22001 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34419
I20260924 04:17:59.342939 22068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34419 every 8 connection(s)
I20260924 04:17:59.343874 22001 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:17:59.345293 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 22001
I20260924 04:17:59.345410 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790223216265354-392-0/minicluster-data/master-0/wal/instance
I20260924 04:17:59.345654 392 external_mini_cluster.cc:1428] Setting key 8540384409fba7efe15ba5a2dbd68cd1
I20260924 04:17:59.346657 22069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:17:59.352061 22069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09c6a03e501d4f02a32588655fe2a09c: Bootstrap starting.
Sep 24 04:17:59 dist-test-slave-p401 krb5kdc[21988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:17:59.354457 22069 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09c6a03e501d4f02a32588655fe2a09c: Neither blocks nor log segments found. Creating new log.
I20260924 04:17:59.355222 22069 log.cc:824] T 00000000000000000000000000000000 P 09c6a03e501d4f02a32588655fe2a09c: Log is configured to *not* fsync() on all Append() calls
I20260924 04:17:59.357282 22069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09c6a03e501d4f02a32588655fe2a09c: No bootstrap required, opened a new log
I20260924 04:17:59.358978 22072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:59.346974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39459 (local address 127.0.98.62:34419)
0924 04:17:59.347409 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:59.347424 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:59.347458 (+ 34us) server_negotiation.cc:407] Connection header received
0924 04:17:59.348079 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:59.348095 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:59.348452 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:59.348767 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:17:59.349525 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:59.350519 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:59.351385 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:59.351848 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:59.352002 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:59.354544 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:17:59.354551 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:17:59.354568 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:17:59.354595 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:17:59.356324 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:17:59.356782 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:17:59.356787 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:17:59.356791 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:17:59.356894 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:17:59.357186 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:17:59.357188 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:17:59.357189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:17:59.357587 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:17:59.357772 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:59.358077 (+ 305us) server_negotiation.cc:299] Negotiation successful
0924 04:17:59.358295 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":158,"threads_started":1}
I20260924 04:17:59.361377 22069 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09c6a03e501d4f02a32588655fe2a09c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09c6a03e501d4f02a32588655fe2a09c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34419 } }
I20260924 04:17:59.361644 22069 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09c6a03e501d4f02a32588655fe2a09c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:17:59.361708 22069 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09c6a03e501d4f02a32588655fe2a09c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09c6a03e501d4f02a32588655fe2a09c, State: Initialized, Role: FOLLOWER
I20260924 04:17:59.362021 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:17:59.362185 22069 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09c6a03e501d4f02a32588655fe2a09c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09c6a03e501d4f02a32588655fe2a09c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34419 } }
I20260924 04:17:59.362322 22069 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09c6a03e501d4f02a32588655fe2a09c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:17:59.362389 22069 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09c6a03e501d4f02a32588655fe2a09c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:17:59.362490 22069 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09c6a03e501d4f02a32588655fe2a09c [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:17:59.363566 22069 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09c6a03e501d4f02a32588655fe2a09c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09c6a03e501d4f02a32588655fe2a09c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34419 } }
I20260924 04:17:59.363929 22069 leader_election.cc:302] T 00000000000000000000000000000000 P 09c6a03e501d4f02a32588655fe2a09c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09c6a03e501d4f02a32588655fe2a09c; no voters:
I20260924 04:17:59.364269 22069 leader_election.cc:288] T 00000000000000000000000000000000 P 09c6a03e501d4f02a32588655fe2a09c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:17:59.364347 22079 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09c6a03e501d4f02a32588655fe2a09c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:17:59.364521 22079 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09c6a03e501d4f02a32588655fe2a09c [term 1 LEADER]: Becoming Leader. State: Replica: 09c6a03e501d4f02a32588655fe2a09c, State: Running, Role: LEADER
I20260924 04:17:59.364835 22079 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09c6a03e501d4f02a32588655fe2a09c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09c6a03e501d4f02a32588655fe2a09c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34419 } }
I20260924 04:17:59.365471 22069 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09c6a03e501d4f02a32588655fe2a09c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:17:59.366401 22081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09c6a03e501d4f02a32588655fe2a09c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09c6a03e501d4f02a32588655fe2a09c. Latest consensus state: current_term: 1 leader_uuid: "09c6a03e501d4f02a32588655fe2a09c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09c6a03e501d4f02a32588655fe2a09c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34419 } } }
I20260924 04:17:59.366533 22081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09c6a03e501d4f02a32588655fe2a09c [sys.catalog]: This master's current role is: LEADER
I20260924 04:17:59.366398 22080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09c6a03e501d4f02a32588655fe2a09c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09c6a03e501d4f02a32588655fe2a09c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09c6a03e501d4f02a32588655fe2a09c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34419 } } }
I20260924 04:17:59.366849 22080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09c6a03e501d4f02a32588655fe2a09c [sys.catalog]: This master's current role is: LEADER
I20260924 04:17:59.366879 22088 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:17:59.369979 22088 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:17:59.375823 22088 catalog_manager.cc:1347] Generated new cluster ID: e943fed00d394a75953605ceba8ddf6d
I20260924 04:17:59.375936 22088 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:17:59.377018 22072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:17:59.363940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37551 (local address 127.0.98.62:34419)
0924 04:17:59.364041 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0924 04:17:59.364045 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:17:59.367525 (+ 3480us) server_negotiation.cc:407] Connection header received
0924 04:17:59.367766 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:17:59.367769 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:17:59.367814 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:17:59.367895 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:17:59.368740 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:59.369266 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:59.370427 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:17:59.370725 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:17:59.370790 (+ 65us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:17:59.372736 (+ 1946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:17:59.372740 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:17:59.372743 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:17:59.372769 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:17:59.374794 (+ 2025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:17:59.375371 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:17:59.375375 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:17:59.375377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:17:59.375463 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:17:59.375896 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:17:59.375899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:17:59.375902 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:17:59.376133 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:17:59.376249 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 04:17:59.376725 (+ 476us) server_negotiation.cc:299] Negotiation successful
0924 04:17:59.376838 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 04:17:59.398289 22088 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:17:59.399327 22088 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:17:59.404803 22088 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09c6a03e501d4f02a32588655fe2a09c: Generated new TSK 0
I20260924 04:17:59.405650 22088 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:17:59.417260 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 22001
W20260924 04:17:59.421674 21986 connection.cc:535] client connection to 127.0.98.62:34419 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:34419): Connection reset by peer (error 104)
2026-09-24T04:17:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1310 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790223216265354-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:17:59 dist-test-slave-p401 krb5kdc[22100](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:17:59 dist-test-slave-p401 krb5kdc[22100](info): set up 2 sockets
Sep 24 04:17:59 dist-test-slave-p401 krb5kdc[22100](info): commencing operation
krb5kdc: starting...
W20260924 04:18:00.355178 392 mini_kdc.cc:121] Time spent starting KDC: real 0.920s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790223216265354-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790223216265354-392-0/krb5kdc/test-user.keytab.
Sep 24 04:18:00 dist-test-slave-p401 krb5kdc[22100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:18:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:18:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:18:00.473637 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790223216265354-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790223216265354-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790223216265354-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39381
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39381
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:18:00.582280 22113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:18:00.582573 22113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:18:00.582645 22113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:18:00.586779 22113 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:18:00.586867 22113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:18:00.586899 22113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:18:00.586926 22113 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:18:00.586956 22113 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:18:00.592491 22113 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790223216265354-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790223216265354-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39381
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790223216265354-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:18:00.593631 22113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:18:00.594707 22113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:18:00.603296 22119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:18:00.603295 22118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:18:00.603343 22121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:18:00.603603 22113 server_base.cc:1034] running on GCE node
I20260924 04:18:00.604208 22113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:18:00.605361 22113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:18:00.606539 22113 hybrid_clock.cc:648] HybridClock initialized: now 1790223480606525 us; error 33 us; skew 500 ppm
Sep 24 04:18:00 dist-test-slave-p401 krb5kdc[22100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223480, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:18:00.609326 22113 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:18:00.610178 22113 webserver.cc:460] Webserver started at http://127.0.98.62:38049/ using document root <none> and password file <none>
I20260924 04:18:00.610697 22113 fs_manager.cc:359] Metadata directory not provided
I20260924 04:18:00.610755 22113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:18:00.610918 22113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:18:00.612957 22113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790223216265354-392-0/minicluster-data/master-0/data/instance:
uuid: "e4cf6fa35e084be8bda7f104b4a452b6"
format_stamp: "Formatted at 2026-09-24 04:18:00 on dist-test-slave-p401"
server_key: "be6d3cd458ebdf0a8ee792b0220e1404"
server_key_iv: "d7585807b0fd6325fa206b7b0e57a138"
server_key_version: "encryptionkey@0"
I20260924 04:18:00.613454 22113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790223216265354-392-0/minicluster-data/master-0/wal/instance:
uuid: "e4cf6fa35e084be8bda7f104b4a452b6"
format_stamp: "Formatted at 2026-09-24 04:18:00 on dist-test-slave-p401"
server_key: "be6d3cd458ebdf0a8ee792b0220e1404"
server_key_iv: "d7585807b0fd6325fa206b7b0e57a138"
server_key_version: "encryptionkey@0"
I20260924 04:18:00.617193 22113 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:18:00.619678 22128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:18:00.620966 22113 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:18:00.621078 22113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790223216265354-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790223216265354-392-0/minicluster-data/master-0/wal
uuid: "e4cf6fa35e084be8bda7f104b4a452b6"
format_stamp: "Formatted at 2026-09-24 04:18:00 on dist-test-slave-p401"
server_key: "be6d3cd458ebdf0a8ee792b0220e1404"
server_key_iv: "d7585807b0fd6325fa206b7b0e57a138"
server_key_version: "encryptionkey@0"
I20260924 04:18:00.621170 22113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790223216265354-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790223216265354-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790223216265354-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:18:00.642222 22113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:18:00.645242 22113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:18:00.645398 22113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:18:00.652195 22113 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39381
I20260924 04:18:00.652252 22180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39381 every 8 connection(s)
I20260924 04:18:00.653152 22113 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:18:00.655974 22181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:18:00.659786 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 22113
I20260924 04:18:00.659914 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790223216265354-392-0/minicluster-data/master-0/wal/instance
I20260924 04:18:00.660189 392 external_mini_cluster.cc:1428] Setting key 944716fe72c1f520a4cdb89a08243e2e
I20260924 04:18:00.661985 22181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4cf6fa35e084be8bda7f104b4a452b6: Bootstrap starting.
I20260924 04:18:00.664445 22181 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4cf6fa35e084be8bda7f104b4a452b6: Neither blocks nor log segments found. Creating new log.
I20260924 04:18:00.665151 22181 log.cc:824] T 00000000000000000000000000000000 P e4cf6fa35e084be8bda7f104b4a452b6: Log is configured to *not* fsync() on all Append() calls
I20260924 04:18:00.667047 22181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4cf6fa35e084be8bda7f104b4a452b6: No bootstrap required, opened a new log
I20260924 04:18:00.669356 22184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:00.661583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42467 (local address 127.0.98.62:39381)
0924 04:18:00.662029 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:00.662047 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:00.662092 (+ 45us) server_negotiation.cc:407] Connection header received
0924 04:18:00.662954 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:00.662977 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:00.663385 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:00.663692 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:00.664104 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:00.665128 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:00.666202 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:00.666659 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:00.666813 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:00.667724 (+ 911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:18:00.667729 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:18:00.667742 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:18:00.668130 (+ 388us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:18:00.668184 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:18:00.668206 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:18:00.668324 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 04:18:00.668519 (+ 195us) server_negotiation.cc:299] Negotiation successful
0924 04:18:00.668647 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":133,"threads_started":1}
I20260924 04:18:00.670774 22181 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4cf6fa35e084be8bda7f104b4a452b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4cf6fa35e084be8bda7f104b4a452b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39381 } }
I20260924 04:18:00.671043 22181 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4cf6fa35e084be8bda7f104b4a452b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:18:00.671144 22181 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4cf6fa35e084be8bda7f104b4a452b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cf6fa35e084be8bda7f104b4a452b6, State: Initialized, Role: FOLLOWER
I20260924 04:18:00.671633 22181 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4cf6fa35e084be8bda7f104b4a452b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4cf6fa35e084be8bda7f104b4a452b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39381 } }
I20260924 04:18:00.671788 22181 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4cf6fa35e084be8bda7f104b4a452b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:18:00.671852 22181 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4cf6fa35e084be8bda7f104b4a452b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:18:00.671972 22181 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4cf6fa35e084be8bda7f104b4a452b6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:18:00.672355 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:18:00.672966 22181 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4cf6fa35e084be8bda7f104b4a452b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4cf6fa35e084be8bda7f104b4a452b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39381 } }
I20260924 04:18:00.673288 22181 leader_election.cc:302] T 00000000000000000000000000000000 P e4cf6fa35e084be8bda7f104b4a452b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4cf6fa35e084be8bda7f104b4a452b6; no voters:
I20260924 04:18:00.673588 22181 leader_election.cc:288] T 00000000000000000000000000000000 P e4cf6fa35e084be8bda7f104b4a452b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:18:00.673696 22190 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4cf6fa35e084be8bda7f104b4a452b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:18:00.673892 22190 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4cf6fa35e084be8bda7f104b4a452b6 [term 1 LEADER]: Becoming Leader. State: Replica: e4cf6fa35e084be8bda7f104b4a452b6, State: Running, Role: LEADER
I20260924 04:18:00.674258 22190 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4cf6fa35e084be8bda7f104b4a452b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4cf6fa35e084be8bda7f104b4a452b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39381 } }
I20260924 04:18:00.674834 22181 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4cf6fa35e084be8bda7f104b4a452b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:18:00.676059 22192 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4cf6fa35e084be8bda7f104b4a452b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4cf6fa35e084be8bda7f104b4a452b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4cf6fa35e084be8bda7f104b4a452b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39381 } } }
I20260924 04:18:00.676177 22192 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4cf6fa35e084be8bda7f104b4a452b6 [sys.catalog]: This master's current role is: LEADER
I20260924 04:18:00.676541 22191 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4cf6fa35e084be8bda7f104b4a452b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4cf6fa35e084be8bda7f104b4a452b6. Latest consensus state: current_term: 1 leader_uuid: "e4cf6fa35e084be8bda7f104b4a452b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4cf6fa35e084be8bda7f104b4a452b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39381 } } }
I20260924 04:18:00.676677 22191 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4cf6fa35e084be8bda7f104b4a452b6 [sys.catalog]: This master's current role is: LEADER
I20260924 04:18:00.676980 22201 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:18:00.679297 22184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:00.674311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43375 (local address 127.0.98.62:39381)
0924 04:18:00.674414 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:00.674418 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:00.674638 (+ 220us) server_negotiation.cc:407] Connection header received
0924 04:18:00.674807 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:00.674810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:00.674855 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:00.674902 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:00.675336 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:00.676104 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:00.677427 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:00.677762 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:00.677873 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:00.678831 (+ 958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:18:00.678834 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:18:00.678836 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:18:00.678847 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:18:00.678870 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:18:00.678880 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:18:00.678952 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 04:18:00.679104 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 04:18:00.679136 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 04:18:00.680974 22201 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:18:00.687350 22201 catalog_manager.cc:1347] Generated new cluster ID: b9393913cf41486bb998f8b36655af2b
I20260924 04:18:00.687426 22201 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:18:00.704491 22201 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:18:00.705863 22201 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:18:00.715044 22201 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4cf6fa35e084be8bda7f104b4a452b6: Generated new TSK 0
I20260924 04:18:00.715744 22201 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:18:00.729168 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 22113
W20260924 04:18:00.733839 22098 connection.cc:535] client connection to 127.0.98.62:39381 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:39381): Connection reset by peer (error 104)
2026-09-24T04:18:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1312 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790223216265354-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:18:00 dist-test-slave-p401 krb5kdc[22212](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:18:00 dist-test-slave-p401 krb5kdc[22212](info): set up 2 sockets
Sep 24 04:18:00 dist-test-slave-p401 krb5kdc[22212](info): commencing operation
krb5kdc: starting...
W20260924 04:18:01.675261 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790223216265354-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790223216265354-392-0/krb5kdc/test-user.keytab.
Sep 24 04:18:01 dist-test-slave-p401 krb5kdc[22212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:18:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:18:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:18:01.793792 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790223216265354-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790223216265354-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790223216265354-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43381
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43381
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:18:01.901762 22225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:18:01.902034 22225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:18:01.902079 22225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:18:01.906085 22225 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:18:01.906147 22225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:18:01.906167 22225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:18:01.906181 22225 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:18:01.906195 22225 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:18:01.911614 22225 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790223216265354-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790223216265354-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43381
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790223216265354-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:18:01.912647 22225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:18:01.913636 22225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:18:01.921857 22233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:18:01.921860 22230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:18:01.921913 22225 server_base.cc:1034] running on GCE node
W20260924 04:18:01.921877 22231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:18:01.922696 22225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:18:01.923833 22225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:18:01.924975 22225 hybrid_clock.cc:648] HybridClock initialized: now 1790223481924960 us; error 36 us; skew 500 ppm
Sep 24 04:18:01 dist-test-slave-p401 krb5kdc[22212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223481, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:18:01.927569 22225 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:18:01.928437 22225 webserver.cc:460] Webserver started at http://127.0.98.62:40149/ using document root <none> and password file <none>
I20260924 04:18:01.928956 22225 fs_manager.cc:359] Metadata directory not provided
I20260924 04:18:01.929039 22225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:18:01.929225 22225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:18:01.931277 22225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790223216265354-392-0/minicluster-data/master-0/data/instance:
uuid: "4e635d96504649ca8b6214ce19bd9c55"
format_stamp: "Formatted at 2026-09-24 04:18:01 on dist-test-slave-p401"
server_key: "89742ac22cb4e68a9ef8ff4fab32ec73"
server_key_iv: "d1c2ac5066be4ee819e6f27db1f06ffb"
server_key_version: "encryptionkey@0"
I20260924 04:18:01.931823 22225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790223216265354-392-0/minicluster-data/master-0/wal/instance:
uuid: "4e635d96504649ca8b6214ce19bd9c55"
format_stamp: "Formatted at 2026-09-24 04:18:01 on dist-test-slave-p401"
server_key: "89742ac22cb4e68a9ef8ff4fab32ec73"
server_key_iv: "d1c2ac5066be4ee819e6f27db1f06ffb"
server_key_version: "encryptionkey@0"
I20260924 04:18:01.935472 22225 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:18:01.937788 22240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:18:01.938946 22225 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:18:01.939054 22225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790223216265354-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790223216265354-392-0/minicluster-data/master-0/wal
uuid: "4e635d96504649ca8b6214ce19bd9c55"
format_stamp: "Formatted at 2026-09-24 04:18:01 on dist-test-slave-p401"
server_key: "89742ac22cb4e68a9ef8ff4fab32ec73"
server_key_iv: "d1c2ac5066be4ee819e6f27db1f06ffb"
server_key_version: "encryptionkey@0"
I20260924 04:18:01.939146 22225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790223216265354-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790223216265354-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790223216265354-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:18:01.943509 22225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:18:01.946436 22225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:18:01.946588 22225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:18:01.953222 22225 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43381
I20260924 04:18:01.953267 22292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43381 every 8 connection(s)
I20260924 04:18:01.954123 22225 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:18:01.956853 22293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:18:01.959623 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 22225
I20260924 04:18:01.959758 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790223216265354-392-0/minicluster-data/master-0/wal/instance
I20260924 04:18:01.959991 392 external_mini_cluster.cc:1428] Setting key a35e00e8069ecca0b4d2d5658118c659
I20260924 04:18:01.962699 22293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e635d96504649ca8b6214ce19bd9c55: Bootstrap starting.
I20260924 04:18:01.965312 22296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:01.961305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60251 (local address 127.0.98.62:43381)
0924 04:18:01.961738 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:01.961761 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:01.961803 (+ 42us) server_negotiation.cc:407] Connection header received
0924 04:18:01.962600 (+ 797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:01.962622 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:01.962997 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:01.963387 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:01.963977 (+ 590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:18:01.963983 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:18:01.963998 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:18:01.964008 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:18:01.964065 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:18:01.964090 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:18:01.964231 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 04:18:01.964297 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 04:18:01.964439 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":118,"threads_started":1}
I20260924 04:18:01.965378 22293 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e635d96504649ca8b6214ce19bd9c55: Neither blocks nor log segments found. Creating new log.
I20260924 04:18:01.966207 22293 log.cc:824] T 00000000000000000000000000000000 P 4e635d96504649ca8b6214ce19bd9c55: Log is configured to *not* fsync() on all Append() calls
I20260924 04:18:01.968292 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:18:01.968415 22293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e635d96504649ca8b6214ce19bd9c55: No bootstrap required, opened a new log
I20260924 04:18:01.972093 22296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:01.970272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52451 (local address 127.0.98.62:43381)
0924 04:18:01.970378 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:01.970381 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:01.970640 (+ 259us) server_negotiation.cc:407] Connection header received
0924 04:18:01.970757 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:01.970760 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:01.970794 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:01.970868 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:01.971653 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:18:01.971656 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:18:01.971658 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:18:01.971668 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:18:01.971694 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:18:01.971706 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:18:01.971776 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 04:18:01.971918 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 04:18:01.971968 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 04:18:01.972815 22293 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e635d96504649ca8b6214ce19bd9c55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e635d96504649ca8b6214ce19bd9c55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43381 } }
I20260924 04:18:01.973145 22293 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e635d96504649ca8b6214ce19bd9c55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:18:01.973214 22293 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e635d96504649ca8b6214ce19bd9c55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e635d96504649ca8b6214ce19bd9c55, State: Initialized, Role: FOLLOWER
I20260924 04:18:01.973639 22293 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e635d96504649ca8b6214ce19bd9c55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e635d96504649ca8b6214ce19bd9c55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43381 } }
I20260924 04:18:01.973773 22293 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e635d96504649ca8b6214ce19bd9c55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:18:01.973842 22293 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e635d96504649ca8b6214ce19bd9c55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:18:01.973943 22293 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e635d96504649ca8b6214ce19bd9c55 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:18:01.974934 22293 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e635d96504649ca8b6214ce19bd9c55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e635d96504649ca8b6214ce19bd9c55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43381 } }
I20260924 04:18:01.975310 22293 leader_election.cc:302] T 00000000000000000000000000000000 P 4e635d96504649ca8b6214ce19bd9c55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e635d96504649ca8b6214ce19bd9c55; no voters:
I20260924 04:18:01.975615 22293 leader_election.cc:288] T 00000000000000000000000000000000 P 4e635d96504649ca8b6214ce19bd9c55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:18:01.975698 22303 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e635d96504649ca8b6214ce19bd9c55 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:18:01.975948 22303 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e635d96504649ca8b6214ce19bd9c55 [term 1 LEADER]: Becoming Leader. State: Replica: 4e635d96504649ca8b6214ce19bd9c55, State: Running, Role: LEADER
I20260924 04:18:01.976343 22303 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e635d96504649ca8b6214ce19bd9c55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e635d96504649ca8b6214ce19bd9c55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43381 } }
I20260924 04:18:01.976904 22293 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e635d96504649ca8b6214ce19bd9c55 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:18:01.977782 22305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e635d96504649ca8b6214ce19bd9c55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e635d96504649ca8b6214ce19bd9c55. Latest consensus state: current_term: 1 leader_uuid: "4e635d96504649ca8b6214ce19bd9c55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e635d96504649ca8b6214ce19bd9c55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43381 } } }
I20260924 04:18:01.977957 22305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e635d96504649ca8b6214ce19bd9c55 [sys.catalog]: This master's current role is: LEADER
I20260924 04:18:01.978252 22304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e635d96504649ca8b6214ce19bd9c55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e635d96504649ca8b6214ce19bd9c55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e635d96504649ca8b6214ce19bd9c55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43381 } } }
I20260924 04:18:01.978372 22312 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:18:01.978936 22304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e635d96504649ca8b6214ce19bd9c55 [sys.catalog]: This master's current role is: LEADER
I20260924 04:18:01.981240 22312 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:18:01.986855 22312 catalog_manager.cc:1347] Generated new cluster ID: ff8f303f7ac241e4bd962f9cdfbdcd66
I20260924 04:18:01.986934 22312 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:18:02.006304 22312 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:18:02.006646 22312 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:18:02.014043 22312 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e635d96504649ca8b6214ce19bd9c55: Generated new TSK 0
I20260924 04:18:02.014982 22312 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:18:02.026988 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 22225
2026-09-24T04:18:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1297 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790223216265354-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:18:02 dist-test-slave-p401 krb5kdc[22324](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:18:02 dist-test-slave-p401 krb5kdc[22324](info): set up 2 sockets
Sep 24 04:18:02 dist-test-slave-p401 krb5kdc[22324](info): commencing operation
krb5kdc: starting...
W20260924 04:18:02.983182 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790223216265354-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790223216265354-392-0/krb5kdc/test-user.keytab.
Sep 24 04:18:03 dist-test-slave-p401 krb5kdc[22324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:18:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:18:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:18:03.099732 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790223216265354-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790223216265354-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790223216265354-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46595
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:18:03.208386 22337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:18:03.208673 22337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:18:03.208714 22337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:18:03.212770 22337 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:18:03.212839 22337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:18:03.212858 22337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:18:03.212873 22337 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:18:03.212888 22337 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:18:03.218389 22337 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790223216265354-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790223216265354-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790223216265354-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:18:03.219501 22337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:18:03.220582 22337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:18:03.228960 22343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:18:03.228960 22345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:18:03.228933 22342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:18:03.229172 22337 server_base.cc:1034] running on GCE node
I20260924 04:18:03.229856 22337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:18:03.230973 22337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:18:03.232125 22337 hybrid_clock.cc:648] HybridClock initialized: now 1790223483232110 us; error 32 us; skew 500 ppm
Sep 24 04:18:03 dist-test-slave-p401 krb5kdc[22324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223483, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:18:03.234727 22337 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:18:03.235903 22337 webserver.cc:460] Webserver started at http://127.0.98.62:46709/ using document root <none> and password file <none>
I20260924 04:18:03.236474 22337 fs_manager.cc:359] Metadata directory not provided
I20260924 04:18:03.236537 22337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:18:03.236742 22337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:18:03.238747 22337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790223216265354-392-0/minicluster-data/master-0/data/instance:
uuid: "791baf664ddc4dea92da30a7ada5393a"
format_stamp: "Formatted at 2026-09-24 04:18:03 on dist-test-slave-p401"
server_key: "92be19176af41a15bab174b9f647f8b7"
server_key_iv: "e07b1328e54888c500465558d0f75b85"
server_key_version: "encryptionkey@0"
I20260924 04:18:03.239315 22337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790223216265354-392-0/minicluster-data/master-0/wal/instance:
uuid: "791baf664ddc4dea92da30a7ada5393a"
format_stamp: "Formatted at 2026-09-24 04:18:03 on dist-test-slave-p401"
server_key: "92be19176af41a15bab174b9f647f8b7"
server_key_iv: "e07b1328e54888c500465558d0f75b85"
server_key_version: "encryptionkey@0"
I20260924 04:18:03.242857 22337 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:18:03.245301 22352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:18:03.246487 22337 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:18:03.246589 22337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790223216265354-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790223216265354-392-0/minicluster-data/master-0/wal
uuid: "791baf664ddc4dea92da30a7ada5393a"
format_stamp: "Formatted at 2026-09-24 04:18:03 on dist-test-slave-p401"
server_key: "92be19176af41a15bab174b9f647f8b7"
server_key_iv: "e07b1328e54888c500465558d0f75b85"
server_key_version: "encryptionkey@0"
I20260924 04:18:03.246686 22337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790223216265354-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790223216265354-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790223216265354-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:18:03.273494 22337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:18:03.276504 22337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:18:03.276678 22337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:18:03.283393 22337 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46595
I20260924 04:18:03.283417 22404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46595 every 8 connection(s)
I20260924 04:18:03.284324 22337 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:18:03.285945 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 22337
I20260924 04:18:03.286055 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790223216265354-392-0/minicluster-data/master-0/wal/instance
I20260924 04:18:03.286347 392 external_mini_cluster.cc:1428] Setting key b894333d40de303f909b5e93dc6dd29d
I20260924 04:18:03.287292 22405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:18:03.293483 22405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 791baf664ddc4dea92da30a7ada5393a: Bootstrap starting.
Sep 24 04:18:03 dist-test-slave-p401 krb5kdc[22324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:18:03.295876 22405 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 791baf664ddc4dea92da30a7ada5393a: Neither blocks nor log segments found. Creating new log.
I20260924 04:18:03.296602 22405 log.cc:824] T 00000000000000000000000000000000 P 791baf664ddc4dea92da30a7ada5393a: Log is configured to *not* fsync() on all Append() calls
I20260924 04:18:03.298440 22405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 791baf664ddc4dea92da30a7ada5393a: No bootstrap required, opened a new log
I20260924 04:18:03.300010 22408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:03.287747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56673 (local address 127.0.98.62:46595)
0924 04:18:03.288118 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:03.288133 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:03.288167 (+ 34us) server_negotiation.cc:407] Connection header received
0924 04:18:03.288794 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:03.288810 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:03.289141 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:03.289416 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:03.290267 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:03.291320 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:03.292497 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:03.292969 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:03.293083 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:03.295713 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:18:03.295719 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:18:03.295733 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:18:03.295762 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:18:03.297329 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:03.297731 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:03.297736 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:03.297741 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:03.297850 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:03.298103 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:03.298106 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:03.298108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:03.298519 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:18:03.298705 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0924 04:18:03.298975 (+ 270us) server_negotiation.cc:299] Negotiation successful
0924 04:18:03.299303 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":102,"threads_started":1}
I20260924 04:18:03.301975 22405 raft_consensus.cc:348] T 00000000000000000000000000000000 P 791baf664ddc4dea92da30a7ada5393a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "791baf664ddc4dea92da30a7ada5393a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46595 } }
I20260924 04:18:03.302229 22405 raft_consensus.cc:374] T 00000000000000000000000000000000 P 791baf664ddc4dea92da30a7ada5393a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:18:03.302297 22405 raft_consensus.cc:729] T 00000000000000000000000000000000 P 791baf664ddc4dea92da30a7ada5393a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 791baf664ddc4dea92da30a7ada5393a, State: Initialized, Role: FOLLOWER
I20260924 04:18:03.302644 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:18:03.302721 22405 consensus_queue.cc:260] T 00000000000000000000000000000000 P 791baf664ddc4dea92da30a7ada5393a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "791baf664ddc4dea92da30a7ada5393a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46595 } }
I20260924 04:18:03.302861 22405 raft_consensus.cc:388] T 00000000000000000000000000000000 P 791baf664ddc4dea92da30a7ada5393a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:18:03.302922 22405 raft_consensus.cc:482] T 00000000000000000000000000000000 P 791baf664ddc4dea92da30a7ada5393a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:18:03.303013 22405 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 791baf664ddc4dea92da30a7ada5393a [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:18:03.303952 22405 raft_consensus.cc:504] T 00000000000000000000000000000000 P 791baf664ddc4dea92da30a7ada5393a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "791baf664ddc4dea92da30a7ada5393a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46595 } }
I20260924 04:18:03.304271 22405 leader_election.cc:302] T 00000000000000000000000000000000 P 791baf664ddc4dea92da30a7ada5393a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 791baf664ddc4dea92da30a7ada5393a; no voters:
I20260924 04:18:03.304554 22405 leader_election.cc:288] T 00000000000000000000000000000000 P 791baf664ddc4dea92da30a7ada5393a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:18:03.304634 22414 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 791baf664ddc4dea92da30a7ada5393a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:18:03.304916 22414 raft_consensus.cc:686] T 00000000000000000000000000000000 P 791baf664ddc4dea92da30a7ada5393a [term 1 LEADER]: Becoming Leader. State: Replica: 791baf664ddc4dea92da30a7ada5393a, State: Running, Role: LEADER
I20260924 04:18:03.305230 22414 consensus_queue.cc:237] T 00000000000000000000000000000000 P 791baf664ddc4dea92da30a7ada5393a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "791baf664ddc4dea92da30a7ada5393a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46595 } }
I20260924 04:18:03.305709 22405 sys_catalog.cc:564] T 00000000000000000000000000000000 P 791baf664ddc4dea92da30a7ada5393a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:18:03.306839 22417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 791baf664ddc4dea92da30a7ada5393a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 791baf664ddc4dea92da30a7ada5393a. Latest consensus state: current_term: 1 leader_uuid: "791baf664ddc4dea92da30a7ada5393a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "791baf664ddc4dea92da30a7ada5393a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46595 } } }
I20260924 04:18:03.306847 22416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 791baf664ddc4dea92da30a7ada5393a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "791baf664ddc4dea92da30a7ada5393a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "791baf664ddc4dea92da30a7ada5393a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46595 } } }
I20260924 04:18:03.307040 22417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 791baf664ddc4dea92da30a7ada5393a [sys.catalog]: This master's current role is: LEADER
I20260924 04:18:03.307029 22416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 791baf664ddc4dea92da30a7ada5393a [sys.catalog]: This master's current role is: LEADER
I20260924 04:18:03.307487 22427 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:18:03.310225 22427 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:18:03.314909 22408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:03.304603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50465 (local address 127.0.98.62:46595)
0924 04:18:03.304712 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:03.304715 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:03.306425 (+ 1710us) server_negotiation.cc:407] Connection header received
0924 04:18:03.306602 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:03.306605 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:03.306651 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:03.306700 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:03.307617 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:03.308367 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:03.309428 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:03.309749 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:03.309820 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:03.311550 (+ 1730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:18:03.311554 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:18:03.311557 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:18:03.311582 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:18:03.313158 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:03.313622 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:03.313626 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:03.313628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:03.313710 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:03.313991 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:03.313994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:03.313996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:03.314253 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:18:03.314390 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 04:18:03.314638 (+ 248us) server_negotiation.cc:299] Negotiation successful
0924 04:18:03.314759 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 04:18:03.315749 22427 catalog_manager.cc:1347] Generated new cluster ID: 5967936ed606467a86cedaed1aa2cdd6
I20260924 04:18:03.315856 22427 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:18:03.332553 22427 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:18:03.333508 22427 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:18:03.342212 22427 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 791baf664ddc4dea92da30a7ada5393a: Generated new TSK 0
I20260924 04:18:03.342901 22427 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:18:03.349432 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 22337
2026-09-24T04:18:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1322 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790223216265354-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:18:03 dist-test-slave-p401 krb5kdc[22436](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:18:03 dist-test-slave-p401 krb5kdc[22436](info): set up 2 sockets
Sep 24 04:18:03 dist-test-slave-p401 krb5kdc[22436](info): commencing operation
krb5kdc: starting...
W20260924 04:18:04.295342 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790223216265354-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790223216265354-392-0/krb5kdc/test-user.keytab.
Sep 24 04:18:04 dist-test-slave-p401 krb5kdc[22436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:18:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:18:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:18:04.415146 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790223216265354-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790223216265354-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790223216265354-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33969
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33969
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:18:04.523459 22449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:18:04.523749 22449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:18:04.523810 22449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:18:04.527854 22449 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:18:04.527926 22449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:18:04.527954 22449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:18:04.527974 22449 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:18:04.527997 22449 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:18:04.533380 22449 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790223216265354-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790223216265354-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790223216265354-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:18:04.534453 22449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:18:04.535480 22449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:18:04.543538 22454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:18:04.543531 22457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:18:04.543530 22455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:18:04.543699 22449 server_base.cc:1034] running on GCE node
I20260924 04:18:04.544392 22449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:18:04.545495 22449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:18:04.546679 22449 hybrid_clock.cc:648] HybridClock initialized: now 1790223484546647 us; error 31 us; skew 500 ppm
Sep 24 04:18:04 dist-test-slave-p401 krb5kdc[22436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223484, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:18:04.553972 22449 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:18:04.554942 22449 webserver.cc:460] Webserver started at http://127.0.98.62:34203/ using document root <none> and password file <none>
I20260924 04:18:04.555562 22449 fs_manager.cc:359] Metadata directory not provided
I20260924 04:18:04.555634 22449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:18:04.555855 22449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:18:04.557932 22449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790223216265354-392-0/minicluster-data/master-0/data/instance:
uuid: "0d85a8aef209486481d7194195b36ce5"
format_stamp: "Formatted at 2026-09-24 04:18:04 on dist-test-slave-p401"
server_key: "ae375e1703094971ed1f3afa59388186"
server_key_iv: "5ca5381f4f589659b040f29b041e9e72"
server_key_version: "encryptionkey@0"
I20260924 04:18:04.558475 22449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790223216265354-392-0/minicluster-data/master-0/wal/instance:
uuid: "0d85a8aef209486481d7194195b36ce5"
format_stamp: "Formatted at 2026-09-24 04:18:04 on dist-test-slave-p401"
server_key: "ae375e1703094971ed1f3afa59388186"
server_key_iv: "5ca5381f4f589659b040f29b041e9e72"
server_key_version: "encryptionkey@0"
I20260924 04:18:04.562107 22449 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.001s
I20260924 04:18:04.564540 22464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:18:04.565748 22449 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:18:04.565856 22449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790223216265354-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790223216265354-392-0/minicluster-data/master-0/wal
uuid: "0d85a8aef209486481d7194195b36ce5"
format_stamp: "Formatted at 2026-09-24 04:18:04 on dist-test-slave-p401"
server_key: "ae375e1703094971ed1f3afa59388186"
server_key_iv: "5ca5381f4f589659b040f29b041e9e72"
server_key_version: "encryptionkey@0"
I20260924 04:18:04.565949 22449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790223216265354-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790223216265354-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790223216265354-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:18:04.587941 22449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:18:04.591473 22449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:18:04.591677 22449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:18:04.598958 22449 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33969
I20260924 04:18:04.598973 22516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33969 every 8 connection(s)
I20260924 04:18:04.600003 22449 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:18:04.601432 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 22449
I20260924 04:18:04.601564 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790223216265354-392-0/minicluster-data/master-0/wal/instance
I20260924 04:18:04.601852 392 external_mini_cluster.cc:1428] Setting key 841d743d2923635bc73510d07312abac
I20260924 04:18:04.603083 22517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:18:04.608469 22517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d85a8aef209486481d7194195b36ce5: Bootstrap starting.
I20260924 04:18:04.610667 22520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:04.603268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56781 (local address 127.0.98.62:33969)
0924 04:18:04.603713 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:04.603729 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:04.603765 (+ 36us) server_negotiation.cc:407] Connection header received
0924 04:18:04.604403 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:04.604420 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:04.604763 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:04.605080 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:04.605464 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:04.606431 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:04.607306 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:04.607767 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:04.607904 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:04.608667 (+ 763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:18:04.608672 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:18:04.608686 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:18:04.609205 (+ 519us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:18:04.609292 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:18:04.609322 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:18:04.609420 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 04:18:04.609561 (+ 141us) server_negotiation.cc:299] Negotiation successful
0924 04:18:04.609776 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":144,"threads_started":1}
I20260924 04:18:04.611060 22517 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d85a8aef209486481d7194195b36ce5: Neither blocks nor log segments found. Creating new log.
I20260924 04:18:04.611929 22517 log.cc:824] T 00000000000000000000000000000000 P 0d85a8aef209486481d7194195b36ce5: Log is configured to *not* fsync() on all Append() calls
I20260924 04:18:04.613488 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:18:04.613840 22517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d85a8aef209486481d7194195b36ce5: No bootstrap required, opened a new log
I20260924 04:18:04.617358 22517 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d85a8aef209486481d7194195b36ce5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d85a8aef209486481d7194195b36ce5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33969 } }
I20260924 04:18:04.617591 22517 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d85a8aef209486481d7194195b36ce5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:18:04.617648 22517 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d85a8aef209486481d7194195b36ce5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d85a8aef209486481d7194195b36ce5, State: Initialized, Role: FOLLOWER
I20260924 04:18:04.618088 22517 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d85a8aef209486481d7194195b36ce5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d85a8aef209486481d7194195b36ce5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33969 } }
I20260924 04:18:04.618252 22517 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d85a8aef209486481d7194195b36ce5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:18:04.618319 22517 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d85a8aef209486481d7194195b36ce5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:18:04.618420 22517 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d85a8aef209486481d7194195b36ce5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:18:04.619575 22520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:04.615341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52369 (local address 127.0.98.62:33969)
0924 04:18:04.615455 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:04.615460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:04.615588 (+ 128us) server_negotiation.cc:407] Connection header received
0924 04:18:04.615680 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:04.615683 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:04.615726 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:04.615802 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:04.616024 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:04.617149 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:04.617975 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:04.618312 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:04.618413 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:04.619155 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:18:04.619159 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:18:04.619162 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:18:04.619174 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:18:04.619205 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:18:04.619218 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:18:04.619318 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 04:18:04.619379 (+ 61us) server_negotiation.cc:299] Negotiation successful
0924 04:18:04.619431 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 04:18:04.619585 22517 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d85a8aef209486481d7194195b36ce5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d85a8aef209486481d7194195b36ce5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33969 } }
I20260924 04:18:04.619937 22517 leader_election.cc:302] T 00000000000000000000000000000000 P 0d85a8aef209486481d7194195b36ce5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d85a8aef209486481d7194195b36ce5; no voters:
I20260924 04:18:04.620299 22517 leader_election.cc:288] T 00000000000000000000000000000000 P 0d85a8aef209486481d7194195b36ce5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:18:04.620419 22527 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d85a8aef209486481d7194195b36ce5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:18:04.620640 22527 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d85a8aef209486481d7194195b36ce5 [term 1 LEADER]: Becoming Leader. State: Replica: 0d85a8aef209486481d7194195b36ce5, State: Running, Role: LEADER
I20260924 04:18:04.620951 22527 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d85a8aef209486481d7194195b36ce5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d85a8aef209486481d7194195b36ce5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33969 } }
I20260924 04:18:04.621467 22517 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d85a8aef209486481d7194195b36ce5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:18:04.621997 22527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d85a8aef209486481d7194195b36ce5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d85a8aef209486481d7194195b36ce5. Latest consensus state: current_term: 1 leader_uuid: "0d85a8aef209486481d7194195b36ce5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d85a8aef209486481d7194195b36ce5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33969 } } }
I20260924 04:18:04.622159 22527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d85a8aef209486481d7194195b36ce5 [sys.catalog]: This master's current role is: LEADER
I20260924 04:18:04.622382 22529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d85a8aef209486481d7194195b36ce5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d85a8aef209486481d7194195b36ce5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d85a8aef209486481d7194195b36ce5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33969 } } }
I20260924 04:18:04.622515 22529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d85a8aef209486481d7194195b36ce5 [sys.catalog]: This master's current role is: LEADER
I20260924 04:18:04.622603 22535 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:18:04.625521 22535 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:18:04.630611 22535 catalog_manager.cc:1347] Generated new cluster ID: dd2b0fd8d5564a8fbcbd4cf7dd7a03ba
I20260924 04:18:04.630662 22535 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:18:04.637902 22535 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:18:04.638815 22535 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:18:04.648270 22535 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d85a8aef209486481d7194195b36ce5: Generated new TSK 0
I20260924 04:18:04.648970 22535 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:18:04.655666 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 22449
2026-09-24T04:18:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1306 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790223216265354-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:18:04 dist-test-slave-p401 krb5kdc[22548](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:18:04 dist-test-slave-p401 krb5kdc[22548](info): set up 2 sockets
Sep 24 04:18:04 dist-test-slave-p401 krb5kdc[22548](info): commencing operation
krb5kdc: starting...
W20260924 04:18:05.623198 392 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790223216265354-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790223216265354-392-0/krb5kdc/test-user.keytab.
Sep 24 04:18:05 dist-test-slave-p401 krb5kdc[22548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:18:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:18:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:18:05.736577 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790223216265354-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790223216265354-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790223216265354-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39879
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:18:05.844026 22561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:18:05.844292 22561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:18:05.844352 22561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:18:05.848272 22561 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:18:05.848325 22561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:18:05.848351 22561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:18:05.848371 22561 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:18:05.848392 22561 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:18:05.853541 22561 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790223216265354-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790223216265354-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790223216265354-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:18:05.854562 22561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:18:05.855569 22561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:18:05.863407 22566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:18:05.863417 22569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:18:05.863474 22561 server_base.cc:1034] running on GCE node
W20260924 04:18:05.863415 22567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:18:05.864199 22561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:18:05.865275 22561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:18:05.866462 22561 hybrid_clock.cc:648] HybridClock initialized: now 1790223485866445 us; error 29 us; skew 500 ppm
Sep 24 04:18:05 dist-test-slave-p401 krb5kdc[22548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223485, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:18:05.869127 22561 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:18:05.870008 22561 webserver.cc:460] Webserver started at http://127.0.98.62:33015/ using document root <none> and password file <none>
I20260924 04:18:05.870532 22561 fs_manager.cc:359] Metadata directory not provided
I20260924 04:18:05.870595 22561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:18:05.870792 22561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:18:05.872861 22561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790223216265354-392-0/minicluster-data/master-0/data/instance:
uuid: "cf29b6375802431c96cef4cd16be3fcb"
format_stamp: "Formatted at 2026-09-24 04:18:05 on dist-test-slave-p401"
server_key: "41eb96bdbe6bba8470e35b2c029f5389"
server_key_iv: "239dc58cf8ee103c8bf9e991f9cec16c"
server_key_version: "encryptionkey@0"
I20260924 04:18:05.873400 22561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790223216265354-392-0/minicluster-data/master-0/wal/instance:
uuid: "cf29b6375802431c96cef4cd16be3fcb"
format_stamp: "Formatted at 2026-09-24 04:18:05 on dist-test-slave-p401"
server_key: "41eb96bdbe6bba8470e35b2c029f5389"
server_key_iv: "239dc58cf8ee103c8bf9e991f9cec16c"
server_key_version: "encryptionkey@0"
I20260924 04:18:05.877113 22561 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:18:05.879499 22576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:18:05.880714 22561 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:18:05.880836 22561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790223216265354-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790223216265354-392-0/minicluster-data/master-0/wal
uuid: "cf29b6375802431c96cef4cd16be3fcb"
format_stamp: "Formatted at 2026-09-24 04:18:05 on dist-test-slave-p401"
server_key: "41eb96bdbe6bba8470e35b2c029f5389"
server_key_iv: "239dc58cf8ee103c8bf9e991f9cec16c"
server_key_version: "encryptionkey@0"
I20260924 04:18:05.880934 22561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790223216265354-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790223216265354-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790223216265354-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:18:05.885407 22561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:18:05.888346 22561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:18:05.888504 22561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:18:05.894862 22561 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39879
I20260924 04:18:05.894896 22628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39879 every 8 connection(s)
I20260924 04:18:05.895746 22561 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:18:05.898396 22629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:18:05.902096 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 22561
I20260924 04:18:05.902212 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790223216265354-392-0/minicluster-data/master-0/wal/instance
I20260924 04:18:05.902465 392 external_mini_cluster.cc:1428] Setting key 6bc1bc97944190ae5ac9710628b579a3
I20260924 04:18:05.903987 22629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf29b6375802431c96cef4cd16be3fcb: Bootstrap starting.
I20260924 04:18:05.906257 22629 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf29b6375802431c96cef4cd16be3fcb: Neither blocks nor log segments found. Creating new log.
I20260924 04:18:05.907011 22629 log.cc:824] T 00000000000000000000000000000000 P cf29b6375802431c96cef4cd16be3fcb: Log is configured to *not* fsync() on all Append() calls
I20260924 04:18:05.907111 22632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:05.903692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60435 (local address 127.0.98.62:39879)
0924 04:18:05.904079 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:05.904093 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:05.904137 (+ 44us) server_negotiation.cc:407] Connection header received
0924 04:18:05.904734 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:05.904751 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:05.905015 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:05.905290 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:05.906023 (+ 733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:18:05.906028 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:18:05.906039 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:18:05.906048 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:18:05.906097 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:18:05.906115 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:18:05.906192 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 04:18:05.906326 (+ 134us) server_negotiation.cc:299] Negotiation successful
0924 04:18:05.906465 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":100,"threads_started":1}
I20260924 04:18:05.908773 22629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf29b6375802431c96cef4cd16be3fcb: No bootstrap required, opened a new log
I20260924 04:18:05.909610 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:18:05.913100 22629 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf29b6375802431c96cef4cd16be3fcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf29b6375802431c96cef4cd16be3fcb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39879 } }
I20260924 04:18:05.913456 22629 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf29b6375802431c96cef4cd16be3fcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:18:05.913537 22629 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf29b6375802431c96cef4cd16be3fcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf29b6375802431c96cef4cd16be3fcb, State: Initialized, Role: FOLLOWER
I20260924 04:18:05.913612 22632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:05.911996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36501 (local address 127.0.98.62:39879)
0924 04:18:05.912097 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:05.912102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:05.912171 (+ 69us) server_negotiation.cc:407] Connection header received
0924 04:18:05.912284 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:05.912287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:05.912334 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:05.912396 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:05.913079 (+ 683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:18:05.913082 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:18:05.913085 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:18:05.913125 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:18:05.913154 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:18:05.913168 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:18:05.913303 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 04:18:05.913388 (+ 85us) server_negotiation.cc:299] Negotiation successful
0924 04:18:05.913427 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 04:18:05.913966 22629 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf29b6375802431c96cef4cd16be3fcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf29b6375802431c96cef4cd16be3fcb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39879 } }
I20260924 04:18:05.914094 22629 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf29b6375802431c96cef4cd16be3fcb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:18:05.914157 22629 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf29b6375802431c96cef4cd16be3fcb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:18:05.914259 22629 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf29b6375802431c96cef4cd16be3fcb [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:18:05.915094 22629 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf29b6375802431c96cef4cd16be3fcb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf29b6375802431c96cef4cd16be3fcb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39879 } }
I20260924 04:18:05.915450 22629 leader_election.cc:302] T 00000000000000000000000000000000 P cf29b6375802431c96cef4cd16be3fcb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf29b6375802431c96cef4cd16be3fcb; no voters:
I20260924 04:18:05.915727 22629 leader_election.cc:288] T 00000000000000000000000000000000 P cf29b6375802431c96cef4cd16be3fcb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:18:05.915827 22639 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf29b6375802431c96cef4cd16be3fcb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:18:05.916043 22639 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf29b6375802431c96cef4cd16be3fcb [term 1 LEADER]: Becoming Leader. State: Replica: cf29b6375802431c96cef4cd16be3fcb, State: Running, Role: LEADER
I20260924 04:18:05.916352 22639 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf29b6375802431c96cef4cd16be3fcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf29b6375802431c96cef4cd16be3fcb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39879 } }
I20260924 04:18:05.916877 22629 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf29b6375802431c96cef4cd16be3fcb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:18:05.917829 22640 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf29b6375802431c96cef4cd16be3fcb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf29b6375802431c96cef4cd16be3fcb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf29b6375802431c96cef4cd16be3fcb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39879 } } }
I20260924 04:18:05.917960 22640 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf29b6375802431c96cef4cd16be3fcb [sys.catalog]: This master's current role is: LEADER
I20260924 04:18:05.918314 22648 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:18:05.918279 22641 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf29b6375802431c96cef4cd16be3fcb [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf29b6375802431c96cef4cd16be3fcb. Latest consensus state: current_term: 1 leader_uuid: "cf29b6375802431c96cef4cd16be3fcb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf29b6375802431c96cef4cd16be3fcb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39879 } } }
I20260924 04:18:05.918432 22641 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf29b6375802431c96cef4cd16be3fcb [sys.catalog]: This master's current role is: LEADER
I20260924 04:18:05.921587 22648 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:18:05.927029 22648 catalog_manager.cc:1347] Generated new cluster ID: c66d1219981448ec9ee482c045b53518
I20260924 04:18:05.927093 22648 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:18:05.938861 22648 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:18:05.939198 22648 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:18:05.952826 22648 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf29b6375802431c96cef4cd16be3fcb: Generated new TSK 0
I20260924 04:18:05.953562 22648 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:18:05.965411 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 22561
2026-09-24T04:18:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1309 ms)
[----------] 18 tests from AuthTokenIssuingTest (23514 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:18:06 dist-test-slave-p401 krb5kdc[22660](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:18:06 dist-test-slave-p401 krb5kdc[22660](info): set up 2 sockets
Sep 24 04:18:06 dist-test-slave-p401 krb5kdc[22660](info): commencing operation
krb5kdc: starting...
W20260924 04:18:06.919924 392 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/krb5kdc/test-user.keytab.
Sep 24 04:18:06 dist-test-slave-p401 krb5kdc[22660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:18:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:18:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:18:07.034225 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38743
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38743,127.0.98.61:42621,127.0.98.60:36213
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:18:07.139894 22673 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 04:18:07.140168 22673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:18:07.140203 22673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:18:07.140264 22673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:18:07.144270 22673 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:18:07.144333 22673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:18:07.144361 22673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:18:07.144382 22673 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:18:07.144411 22673 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:18:07.149482 22673 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38743,127.0.98.61:42621,127.0.98.60:36213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:18:07.150462 22673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:18:07.151460 22673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:18:07.159632 22678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:18:07.159693 22681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:18:07.159704 22673 server_base.cc:1034] running on GCE node
W20260924 04:18:07.159636 22679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:18:07.160441 22673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:18:07.161538 22673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:18:07.162695 22673 hybrid_clock.cc:648] HybridClock initialized: now 1790223487162667 us; error 41 us; skew 500 ppm
Sep 24 04:18:07 dist-test-slave-p401 krb5kdc[22660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223487, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:18:07.165382 22673 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:18:07.166229 22673 webserver.cc:460] Webserver started at http://127.0.98.62:45905/ using document root <none> and password file <none>
I20260924 04:18:07.166728 22673 fs_manager.cc:359] Metadata directory not provided
I20260924 04:18:07.166790 22673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:18:07.166955 22673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:18:07.168955 22673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/minicluster-data/master-0/data/instance:
uuid: "eb5bcf4217704def817f3d9dd941a6ea"
format_stamp: "Formatted at 2026-09-24 04:18:07 on dist-test-slave-p401"
server_key: "a05bfe5fd7614f5a1bca26352ef6963e"
server_key_iv: "8b24d51bda2f799f39a3a56c1d5e10d9"
server_key_version: "encryptionkey@0"
I20260924 04:18:07.169487 22673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/minicluster-data/master-0/wal/instance:
uuid: "eb5bcf4217704def817f3d9dd941a6ea"
format_stamp: "Formatted at 2026-09-24 04:18:07 on dist-test-slave-p401"
server_key: "a05bfe5fd7614f5a1bca26352ef6963e"
server_key_iv: "8b24d51bda2f799f39a3a56c1d5e10d9"
server_key_version: "encryptionkey@0"
I20260924 04:18:07.173111 22673 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:18:07.175451 22688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:18:07.176657 22673 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:18:07.176793 22673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/minicluster-data/master-0/wal
uuid: "eb5bcf4217704def817f3d9dd941a6ea"
format_stamp: "Formatted at 2026-09-24 04:18:07 on dist-test-slave-p401"
server_key: "a05bfe5fd7614f5a1bca26352ef6963e"
server_key_iv: "8b24d51bda2f799f39a3a56c1d5e10d9"
server_key_version: "encryptionkey@0"
I20260924 04:18:07.176887 22673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:18:07.191576 22673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:18:07.194603 22673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:18:07.194787 22673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:18:07.201579 22673 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38743
I20260924 04:18:07.201903 22740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38743 every 8 connection(s)
I20260924 04:18:07.202560 22673 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:18:07.205413 22741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:18:07.209395 22741 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38743 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:18:07.210089 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 22673
I20260924 04:18:07.210203 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/minicluster-data/master-0/wal/instance
I20260924 04:18:07.210444 392 external_mini_cluster.cc:1428] Setting key 8a71d475fd4b657031e00c1f04dcbc14
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 24 04:18:07 dist-test-slave-p401 krb5kdc[22660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223487, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:18:07.227488 22745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:07.211876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.0.1:45524)
0924 04:18:07.212497 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0924 04:18:07.212536 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 04:18:07.213388 (+ 852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:18:07.214056 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:18:07.214061 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:18:07.214499 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:18:07.214782 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:18:07.214800 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:18:07.216034 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:18:07.216038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:18:07.217209 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:18:07.217222 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:18:07.217911 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:18:07.217916 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:18:07.218043 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:07.218631 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:18:07.218647 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:18:07.222282 (+ 3635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:18:07.224493 (+ 2211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:18:07.224502 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:18:07.224514 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:18:07.224730 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:18:07.225026 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:18:07.225029 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:18:07.225031 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:18:07.225147 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:18:07.225726 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:18:07.225733 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:18:07.225874 (+ 141us) client_negotiation.cc:769] Sending connection context
0924 04:18:07.226026 (+ 152us) client_negotiation.cc:240] Negotiation successful
0924 04:18:07.226419 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"mutex_wait_us":90,"spinlock_wait_cycles":46976,"thread_start_us":193,"threads_started":1}
I20260924 04:18:07.227488 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:07.211876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45524 (local address 127.0.98.62:38743)
0924 04:18:07.212350 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:07.212369 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:07.212638 (+ 269us) server_negotiation.cc:407] Connection header received
0924 04:18:07.213681 (+ 1043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:07.213685 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:07.213879 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:07.214041 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:07.214947 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:07.215905 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:07.217351 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:07.217740 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:07.217863 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:07.222481 (+ 4618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:18:07.222487 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:18:07.222500 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:18:07.222526 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:18:07.224383 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:07.224833 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:07.224838 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:07.224839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:07.224943 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:07.225227 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:07.225230 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:07.225232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:07.225621 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:18:07.225748 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 04:18:07.226096 (+ 348us) server_negotiation.cc:299] Negotiation successful
0924 04:18:07.226359 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":323,"thread_start_us":135,"threads_started":1}
I20260924 04:18:07.230010 22741 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42621 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:18:07.230816 22745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:07.230525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42621 (local address 127.0.0.1:37572)
0924 04:18:07.230666 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 04:18:07.230743 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260924 04:18:07.231037 22689 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42621: connect: Connection refused (error 111)
I20260924 04:18:07.231896 22745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:07.231673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42621 (local address 127.0.0.1:37582)
0924 04:18:07.231779 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0924 04:18:07.231831 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 04:18:07.232034 22741 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42621: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42621: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260924 04:18:07.239099 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42621
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38743,127.0.98.61:42621,127.0.98.60:36213
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:18:07.281296 22741 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42621 } attempt: 1
I20260924 04:18:07.282254 22745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:07.281922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42621 (local address 127.0.0.1:37590)
0924 04:18:07.282068 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 04:18:07.282130 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 04:18:07.283066 22745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:07.282781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42621 (local address 127.0.0.1:37600)
0924 04:18:07.282924 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 04:18:07.282994 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 04:18:07.283222 22741 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42621: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42621: connect: Connection refused (error 111)
W20260924 04:18:07.348848 22748 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 04:18:07.349152 22748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:18:07.349188 22748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:18:07.349251 22748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 04:18:07.351451 22741 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42621 } attempt: 2
I20260924 04:18:07.352262 22745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:07.351976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42621 (local address 127.0.0.1:37614)
0924 04:18:07.352122 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 04:18:07.352187 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 04:18:07.352945 22745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:07.352726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42621 (local address 127.0.0.1:37624)
0924 04:18:07.352841 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 04:18:07.352889 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260924 04:18:07.353053 22741 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42621: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42621: connect: Connection refused (error 111)
W20260924 04:18:07.353525 22748 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:18:07.353580 22748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:18:07.353616 22748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:18:07.353646 22748 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:18:07.353678 22748 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:18:07.359054 22748 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38743,127.0.98.61:42621,127.0.98.60:36213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:18:07.360193 22748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:18:07.361209 22748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:18:07.369383 22754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:18:07.369417 22753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:18:07.369395 22756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:18:07.369711 22748 server_base.cc:1034] running on GCE node
I20260924 04:18:07.370221 22748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:18:07.371326 22748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:18:07.372493 22748 hybrid_clock.cc:648] HybridClock initialized: now 1790223487372463 us; error 40 us; skew 500 ppm
Sep 24 04:18:07 dist-test-slave-p401 krb5kdc[22660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223487, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:18:07.375044 22748 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260924 04:18:07.375994 22748 webserver.cc:460] Webserver started at http://127.0.98.61:41881/ using document root <none> and password file <none>
I20260924 04:18:07.376526 22748 fs_manager.cc:359] Metadata directory not provided
I20260924 04:18:07.376590 22748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:18:07.376783 22748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:18:07.378759 22748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/minicluster-data/master-1/data/instance:
uuid: "0584bc9bb958458cb5b9200413d875b1"
format_stamp: "Formatted at 2026-09-24 04:18:07 on dist-test-slave-p401"
server_key: "822ac8184ab16d98da98b8b167bfa43c"
server_key_iv: "fd9d381a4a97a0ddf7d0a3ef7f1fb933"
server_key_version: "encryptionkey@0"
I20260924 04:18:07.379335 22748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/minicluster-data/master-1/wal/instance:
uuid: "0584bc9bb958458cb5b9200413d875b1"
format_stamp: "Formatted at 2026-09-24 04:18:07 on dist-test-slave-p401"
server_key: "822ac8184ab16d98da98b8b167bfa43c"
server_key_iv: "fd9d381a4a97a0ddf7d0a3ef7f1fb933"
server_key_version: "encryptionkey@0"
I20260924 04:18:07.382992 22748 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:18:07.385686 22763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:18:07.387017 22748 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:18:07.387146 22748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/minicluster-data/master-1/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/minicluster-data/master-1/wal
uuid: "0584bc9bb958458cb5b9200413d875b1"
format_stamp: "Formatted at 2026-09-24 04:18:07 on dist-test-slave-p401"
server_key: "822ac8184ab16d98da98b8b167bfa43c"
server_key_iv: "fd9d381a4a97a0ddf7d0a3ef7f1fb933"
server_key_version: "encryptionkey@0"
I20260924 04:18:07.387323 22748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:18:07.398548 22748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:18:07.401880 22748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:18:07.402077 22748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:18:07.409193 22748 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:42621
I20260924 04:18:07.409543 22815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:42621 every 8 connection(s)
I20260924 04:18:07.410210 22748 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/minicluster-data/master-1/data/info.pb
I20260924 04:18:07.413175 22816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:18:07.415395 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 22748
I20260924 04:18:07.415518 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/minicluster-data/master-1/wal/instance
I20260924 04:18:07.415805 392 external_mini_cluster.cc:1428] Setting key a800e232609b47b2f0b2929b4d958e16
I20260924 04:18:07.417328 22816 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38743 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 24 04:18:07 dist-test-slave-p401 krb5kdc[22660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223487, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:18:07.431775 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:07.419315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45526 (local address 127.0.98.62:38743)
0924 04:18:07.419483 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:07.419488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:07.420366 (+ 878us) server_negotiation.cc:407] Connection header received
0924 04:18:07.421359 (+ 993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:07.421364 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:07.421429 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:07.421538 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:07.422889 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:07.423537 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:07.424718 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:07.425040 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:07.425151 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:07.428115 (+ 2964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:18:07.428118 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:18:07.428120 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:18:07.428149 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:18:07.429510 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:07.430119 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:07.430121 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:07.430123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:07.430193 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:07.430558 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:07.430560 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:07.430561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:07.430727 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:18:07.430804 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 04:18:07.431532 (+ 728us) server_negotiation.cc:299] Negotiation successful
0924 04:18:07.431646 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:18:07.432963 22819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:07.419687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.0.1:45526)
0924 04:18:07.420182 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0924 04:18:07.420238 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 04:18:07.421142 (+ 904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:18:07.421766 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:18:07.421773 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:18:07.422153 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:18:07.422726 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:18:07.422745 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:18:07.423691 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:18:07.423695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:18:07.424592 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:18:07.424601 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:18:07.425204 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:18:07.425208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:18:07.425413 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:07.426029 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:18:07.426049 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:18:07.427899 (+ 1850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:18:07.429633 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:18:07.429640 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:18:07.429659 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:18:07.430025 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:18:07.430275 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:18:07.430278 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:18:07.430280 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:18:07.430474 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:18:07.430820 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:18:07.430828 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:18:07.431197 (+ 369us) client_negotiation.cc:769] Sending connection context
0924 04:18:07.431469 (+ 272us) client_negotiation.cc:240] Negotiation successful
0924 04:18:07.431838 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":115,"threads_started":1}
I20260924 04:18:07.434370 22816 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42621 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 04:18:07 dist-test-slave-p401 krb5kdc[22660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223487, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260924 04:18:07.442624 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36213
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38743,127.0.98.61:42621,127.0.98.60:36213
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:18:07.442915 22819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:07.434828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42621 (local address 127.0.0.1:37626)
0924 04:18:07.434961 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 04:18:07.434975 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:18:07.435053 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:18:07.435724 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:18:07.435727 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:18:07.435851 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:18:07.435938 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:18:07.435943 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:18:07.436852 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:18:07.436855 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:18:07.437411 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:18:07.437420 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:18:07.437957 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:18:07.437960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:18:07.438029 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:07.438421 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:18:07.438434 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:18:07.439692 (+ 1258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:18:07.441447 (+ 1755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:18:07.441449 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:18:07.441451 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:18:07.441645 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:18:07.441906 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:18:07.441909 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:18:07.441910 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:18:07.441976 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:18:07.442433 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:18:07.442436 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:18:07.442535 (+ 99us) client_negotiation.cc:769] Sending connection context
0924 04:18:07.442638 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 04:18:07.442768 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 04:18:07.443107 22821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:07.434984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37626 (local address 127.0.98.61:42621)
0924 04:18:07.435365 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:07.435374 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:07.435392 (+ 18us) server_negotiation.cc:407] Connection header received
0924 04:18:07.435425 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:07.435429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:07.435588 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:07.435698 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:07.436021 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:07.436742 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:07.437499 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:07.437860 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:07.437934 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:07.439838 (+ 1904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:18:07.439847 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:18:07.439859 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:18:07.439879 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:18:07.441341 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:07.441741 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:07.441746 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:07.441748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:07.441819 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:07.442058 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:07.442061 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:07.442063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:07.442332 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:18:07.442437 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 04:18:07.442711 (+ 274us) server_negotiation.cc:299] Negotiation successful
0924 04:18:07.442891 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":166,"threads_started":1}
I20260924 04:18:07.444212 22741 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42621 } attempt: 3
I20260924 04:18:07.445206 22816 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36213 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:18:07.446136 22819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:07.445882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36213 (local address 127.0.0.1:56724)
0924 04:18:07.445998 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 04:18:07.446072 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 04:18:07.446390 22764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36213: connect: Connection refused (error 111)
I20260924 04:18:07.447165 22819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:07.446982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36213 (local address 127.0.0.1:56730)
0924 04:18:07.447072 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0924 04:18:07.447116 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 04:18:07.447389 22816 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36213: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36213: connect: Connection refused (error 111)
Sep 24 04:18:07 dist-test-slave-p401 krb5kdc[22660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223487, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 04:18:07.452714 22821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:07.445036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37642 (local address 127.0.98.61:42621)
0924 04:18:07.445151 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:07.445155 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:07.445167 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:18:07.445194 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:07.445196 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:07.445230 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:07.445303 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:07.445745 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:07.446456 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:07.447680 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:07.447906 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:07.447982 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:07.449987 (+ 2005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:18:07.449989 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:18:07.449991 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:18:07.450012 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:18:07.451282 (+ 1270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:07.451713 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:07.451716 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:07.451717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:07.451782 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:07.452043 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:07.452045 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:07.452046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:07.452195 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:18:07.452261 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 04:18:07.452528 (+ 267us) server_negotiation.cc:299] Negotiation successful
0924 04:18:07.452615 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 04:18:07.452811 22745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:07.444676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42621 (local address 127.0.0.1:37642)
0924 04:18:07.444812 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 04:18:07.444827 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:18:07.444912 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:18:07.445347 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:18:07.445350 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:18:07.445523 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:18:07.445630 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:18:07.445636 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:18:07.446607 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:18:07.446611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:18:07.447572 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:18:07.447583 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:18:07.448008 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:18:07.448011 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:18:07.448129 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:07.448534 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:18:07.448549 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:18:07.449850 (+ 1301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:18:07.451384 (+ 1534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:18:07.451387 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:18:07.451389 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:18:07.451628 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:18:07.451857 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:18:07.451860 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:18:07.451862 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:18:07.451967 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:18:07.452275 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:18:07.452278 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:18:07.452385 (+ 107us) client_negotiation.cc:769] Sending connection context
0924 04:18:07.452496 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 04:18:07.452634 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 04:18:07.453971 22741 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36213 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:18:07.454617 22745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:07.454373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36213 (local address 127.0.0.1:56738)
0924 04:18:07.454487 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 04:18:07.454543 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 04:18:07.455363 22745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:07.455106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36213 (local address 127.0.0.1:56754)
0924 04:18:07.455260 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 04:18:07.455307 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260924 04:18:07.455489 22741 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36213: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36213: connect: Connection refused (error 111)
I20260924 04:18:07.486640 22741 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36213 } attempt: 1
I20260924 04:18:07.487514 22745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:07.487152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36213 (local address 127.0.0.1:56760)
0924 04:18:07.487347 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 04:18:07.487417 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 04:18:07.488204 22745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:07.487958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36213 (local address 127.0.0.1:56774)
0924 04:18:07.488074 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 04:18:07.488130 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 04:18:07.488353 22741 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36213: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36213: connect: Connection refused (error 111)
I20260924 04:18:07.496647 22816 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36213 } attempt: 1
I20260924 04:18:07.497493 22819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:07.497186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36213 (local address 127.0.0.1:56778)
0924 04:18:07.497354 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 04:18:07.497414 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 04:18:07.498116 22819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:07.497924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36213 (local address 127.0.0.1:56780)
0924 04:18:07.498019 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0924 04:18:07.498062 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 04:18:07.498224 22816 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36213: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36213: connect: Connection refused (error 111)
W20260924 04:18:07.556834 22822 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 04:18:07.557127 22822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:18:07.557163 22822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:18:07.557238 22822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:18:07.561251 22822 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:18:07.561316 22822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:18:07.561342 22822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:18:07.561362 22822 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:18:07.561383 22822 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:18:07.563504 22741 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36213 } attempt: 2
I20260924 04:18:07.564265 22745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:07.563988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36213 (local address 127.0.0.1:56794)
0924 04:18:07.564126 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 04:18:07.564186 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 04:18:07.564977 22745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:07.564789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36213 (local address 127.0.0.1:56802)
0924 04:18:07.564880 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0924 04:18:07.564923 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 04:18:07.565105 22741 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36213: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36213: connect: Connection refused (error 111)
I20260924 04:18:07.566358 22816 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36213 } attempt: 2
I20260924 04:18:07.566931 22819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:07.566703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36213 (local address 127.0.0.1:56804)
0924 04:18:07.566824 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 04:18:07.566871 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 04:18:07.567121 22822 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38743,127.0.98.61:42621,127.0.98.60:36213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:18:07.567530 22819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:07.567356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36213 (local address 127.0.0.1:56814)
0924 04:18:07.567434 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0924 04:18:07.567477 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260924 04:18:07.567631 22816 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36213: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36213: connect: Connection refused (error 111)
I20260924 04:18:07.568225 22822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:18:07.569183 22822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:18:07.577044 22829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:18:07.577123 22828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:18:07.577129 22822 server_base.cc:1034] running on GCE node
W20260924 04:18:07.577327 22831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:18:07.577806 22822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:18:07.578842 22822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:18:07.579988 22822 hybrid_clock.cc:648] HybridClock initialized: now 1790223487579966 us; error 33 us; skew 500 ppm
Sep 24 04:18:07 dist-test-slave-p401 krb5kdc[22660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223487, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:18:07.582521 22822 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260924 04:18:07.583462 22822 webserver.cc:460] Webserver started at http://127.0.98.60:37607/ using document root <none> and password file <none>
I20260924 04:18:07.584004 22822 fs_manager.cc:359] Metadata directory not provided
I20260924 04:18:07.584070 22822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:18:07.584271 22822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:18:07.586293 22822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/minicluster-data/master-2/data/instance:
uuid: "a09394d590524a0dba6ee7f6c6ce475d"
format_stamp: "Formatted at 2026-09-24 04:18:07 on dist-test-slave-p401"
server_key: "a17a1f521ab3a44d96d9d0576bde599d"
server_key_iv: "1b4d4a87e124d824cb8d7aaf95a7ce11"
server_key_version: "encryptionkey@0"
I20260924 04:18:07.586833 22822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/minicluster-data/master-2/wal/instance:
uuid: "a09394d590524a0dba6ee7f6c6ce475d"
format_stamp: "Formatted at 2026-09-24 04:18:07 on dist-test-slave-p401"
server_key: "a17a1f521ab3a44d96d9d0576bde599d"
server_key_iv: "1b4d4a87e124d824cb8d7aaf95a7ce11"
server_key_version: "encryptionkey@0"
I20260924 04:18:07.590406 22822 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 04:18:07.592862 22838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:18:07.594064 22822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:18:07.594188 22822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/minicluster-data/master-2/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/minicluster-data/master-2/wal
uuid: "a09394d590524a0dba6ee7f6c6ce475d"
format_stamp: "Formatted at 2026-09-24 04:18:07 on dist-test-slave-p401"
server_key: "a17a1f521ab3a44d96d9d0576bde599d"
server_key_iv: "1b4d4a87e124d824cb8d7aaf95a7ce11"
server_key_version: "encryptionkey@0"
I20260924 04:18:07.594285 22822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:18:07.618460 22822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:18:07.621619 22822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:18:07.621807 22822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:18:07.628621 22822 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:36213
I20260924 04:18:07.628877 22890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:36213 every 8 connection(s)
I20260924 04:18:07.629596 22822 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/minicluster-data/master-2/data/info.pb
I20260924 04:18:07.632402 22891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:18:07.636243 22891 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38743 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:18:07.639482 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 22822
I20260924 04:18:07.639582 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790223216265354-392-0/minicluster-data/master-2/wal/instance
I20260924 04:18:07.639820 392 external_mini_cluster.cc:1428] Setting key 8b50357830998e67bcf3fa7d41f473b7
Sep 24 04:18:07 dist-test-slave-p401 krb5kdc[22660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223487, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 24 04:18:07 dist-test-slave-p401 krb5kdc[22660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 04:18:07.649367 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:07.638074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45534 (local address 127.0.98.62:38743)
0924 04:18:07.638219 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:07.638223 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:07.639022 (+ 799us) server_negotiation.cc:407] Connection header received
0924 04:18:07.640031 (+ 1009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:07.640034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:07.640088 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:07.640160 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:07.641399 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:07.641887 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:07.642955 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:07.643156 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:07.643267 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:07.645828 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:18:07.645832 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:18:07.645834 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:18:07.645858 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:18:07.647325 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:07.647793 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:07.647796 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:07.647798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:07.647882 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:07.648169 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:07.648172 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:07.648174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:07.648379 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:18:07.648463 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 04:18:07.649062 (+ 599us) server_negotiation.cc:299] Negotiation successful
0924 04:18:07.649185 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 04:18:07.650224 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:07.638424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.0.1:45534)
0924 04:18:07.638877 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0924 04:18:07.638920 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 04:18:07.639833 (+ 913us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:18:07.640344 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:18:07.640353 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:18:07.640790 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:18:07.641268 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:18:07.641280 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:18:07.641982 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:18:07.641985 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:18:07.642829 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:18:07.642839 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:18:07.643326 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:18:07.643330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:18:07.643484 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:07.644053 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:18:07.644073 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:18:07.645699 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:18:07.647423 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:18:07.647431 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:18:07.647440 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:18:07.647698 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:18:07.647964 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:18:07.647966 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:18:07.647967 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:18:07.648088 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:18:07.648475 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:18:07.648480 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:18:07.648773 (+ 293us) client_negotiation.cc:769] Sending connection context
0924 04:18:07.648982 (+ 209us) client_negotiation.cc:240] Negotiation successful
0924 04:18:07.649279 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":36,"spinlock_wait_cycles":768,"thread_start_us":119,"threads_started":1}
I20260924 04:18:07.650223 22895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:07.640411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56822 (local address 127.0.98.60:36213)
0924 04:18:07.640767 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:07.640779 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:07.640802 (+ 23us) server_negotiation.cc:407] Connection header received
0924 04:18:07.640906 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:07.640912 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:07.641075 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:07.641211 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:07.641918 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:07.642795 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:07.643677 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:07.644115 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:07.644216 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:07.646533 (+ 2317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:18:07.646540 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:18:07.646554 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:18:07.646576 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:18:07.648090 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:07.648485 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:07.648491 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:07.648493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:07.648567 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:07.648813 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:07.648815 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:07.648816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:07.649041 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:18:07.649253 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0924 04:18:07.649423 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 04:18:07.649574 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":264,"thread_start_us":123,"threads_started":1}
I20260924 04:18:07.651377 22891 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42621 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 04:18:07 dist-test-slave-p401 krb5kdc[22660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223487, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 24 04:18:07 dist-test-slave-p401 krb5kdc[22660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:18:07.658753 22816 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36213 } attempt: 3
I20260924 04:18:07.659132 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:07.651798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42621 (local address 127.0.0.1:37650)
0924 04:18:07.651900 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0924 04:18:07.651923 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 04:18:07.651987 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:18:07.652269 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:18:07.652271 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:18:07.652391 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:18:07.652476 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:18:07.652480 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:18:07.653228 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:18:07.653231 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:18:07.653764 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:18:07.653771 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:18:07.654154 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:18:07.654157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:18:07.654253 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:07.654656 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:18:07.654667 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:18:07.655883 (+ 1216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:18:07.657576 (+ 1693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:18:07.657578 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:18:07.657580 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:18:07.657829 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:18:07.658108 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:18:07.658111 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:18:07.658113 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:18:07.658203 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:18:07.658633 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:18:07.658636 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:18:07.658735 (+ 99us) client_negotiation.cc:769] Sending connection context
0924 04:18:07.658849 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 04:18:07.658977 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 04:18:07.659170 22821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:07.651958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37650 (local address 127.0.98.61:42621)
0924 04:18:07.652093 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:07.652096 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:07.652107 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:18:07.652143 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:07.652145 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:07.652180 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:07.652242 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:07.652571 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:07.653125 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:07.653864 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:07.654067 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:07.654148 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:07.656017 (+ 1869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:18:07.656021 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:18:07.656023 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:18:07.656051 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:18:07.657460 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:07.657928 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:07.657932 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:07.657935 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:07.658024 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:07.658345 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:07.658347 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:07.658348 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:07.658543 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:18:07.658635 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 04:18:07.658917 (+ 282us) server_negotiation.cc:299] Negotiation successful
0924 04:18:07.659032 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 04:18:07.659840 22891 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36213 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:18:07.662084 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:07.652731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45536 (local address 127.0.98.62:38743)
0924 04:18:07.652832 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:07.652836 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:07.652940 (+ 104us) server_negotiation.cc:407] Connection header received
0924 04:18:07.653037 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:07.653040 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:07.653078 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:07.653139 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:07.653724 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:07.654195 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:07.654916 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:07.655163 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:07.655302 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:07.657674 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:18:07.657678 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:18:07.657681 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:18:07.657705 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:18:07.659888 (+ 2183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:07.660394 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:07.660397 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:07.660399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:07.660486 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:07.660771 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:07.660774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:07.660776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:07.660995 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:18:07.661229 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0924 04:18:07.661826 (+ 597us) server_negotiation.cc:299] Negotiation successful
0924 04:18:07.661936 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
Sep 24 04:18:07 dist-test-slave-p401 krb5kdc[22660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223487, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 04:18:07.664449 22741 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36213 } attempt: 3
Sep 24 04:18:07 dist-test-slave-p401 krb5kdc[22660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223487, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 04:18:07.667861 22819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:07.659123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36213 (local address 127.0.0.1:56830)
0924 04:18:07.659373 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0924 04:18:07.659385 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:18:07.659449 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:18:07.659975 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:18:07.659978 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:18:07.660132 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:18:07.660247 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:18:07.660255 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:18:07.661268 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:18:07.661271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:18:07.661983 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:18:07.661993 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:18:07.662534 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:18:07.662537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:18:07.662615 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:07.662993 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:18:07.663006 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:18:07.664399 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:18:07.666333 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:18:07.666336 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:18:07.666338 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:18:07.666567 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:18:07.666865 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:18:07.666867 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:18:07.666869 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:18:07.666954 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:18:07.667386 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:18:07.667389 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:18:07.667477 (+ 88us) client_negotiation.cc:769] Sending connection context
0924 04:18:07.667560 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 04:18:07.667684 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186}
I20260924 04:18:07.668448 22895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:07.659578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56830 (local address 127.0.98.60:36213)
0924 04:18:07.659779 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:07.659783 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:07.659796 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:18:07.659826 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:07.659828 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:07.659865 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:07.659939 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:07.660386 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:07.661141 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:07.662118 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:07.662430 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:07.662517 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:07.664541 (+ 2024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:18:07.664545 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:18:07.664548 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:18:07.664570 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:18:07.666203 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:07.666683 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:07.666686 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:07.666688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:07.666754 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:07.667057 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:07.667060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:07.667062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:07.667282 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:18:07.668112 (+ 830us) server_negotiation.cc:1035] Waiting for connection context
0924 04:18:07.668177 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 04:18:07.668300 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":134}
Sep 24 04:18:07 dist-test-slave-p401 krb5kdc[22660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 04:18:07.669509 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:07.660540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36213 (local address 127.0.0.1:56842)
0924 04:18:07.660618 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0924 04:18:07.660630 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:18:07.660691 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:18:07.661112 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:18:07.661115 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:18:07.661268 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:18:07.661328 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:18:07.661333 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:18:07.662785 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:18:07.662788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:18:07.663533 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:18:07.663542 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:18:07.664070 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:18:07.664074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:18:07.664153 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:07.664954 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:18:07.664968 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:18:07.666294 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:18:07.668083 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:18:07.668086 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:18:07.668088 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:18:07.668409 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:18:07.668657 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:18:07.668660 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:18:07.668662 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:18:07.668732 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:18:07.669077 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:18:07.669079 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:18:07.669157 (+ 78us) client_negotiation.cc:769] Sending connection context
0924 04:18:07.669235 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 04:18:07.669354 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260924 04:18:07.669565 22896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:07.660665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56842 (local address 127.0.98.60:36213)
0924 04:18:07.660939 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:07.660946 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:07.660955 (+ 9us) server_negotiation.cc:407] Connection header received
0924 04:18:07.660978 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:07.660980 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:07.661009 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:07.661079 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:07.661420 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:07.662671 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:07.663632 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:07.663951 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:07.664058 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:07.666429 (+ 2371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:18:07.666433 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:18:07.666435 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:18:07.666455 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:18:07.667982 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:07.668500 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:07.668502 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:07.668504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:07.668573 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:07.668803 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:07.668807 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:07.668809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:07.668991 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:18:07.669076 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 04:18:07.669285 (+ 209us) server_negotiation.cc:299] Negotiation successful
0924 04:18:07.669404 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":225,"thread_start_us":102,"threads_started":1}
Sep 24 04:18:07 dist-test-slave-p401 krb5kdc[22660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223487, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 04:18:07.672943 22891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a09394d590524a0dba6ee7f6c6ce475d: Bootstrap starting.
I20260924 04:18:07.673537 22816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0584bc9bb958458cb5b9200413d875b1: Bootstrap starting.
I20260924 04:18:07.674412 22745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:07.664801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36213 (local address 127.0.0.1:56846)
0924 04:18:07.664932 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 04:18:07.664945 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:18:07.665012 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:18:07.666274 (+ 1262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:18:07.666277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:18:07.666445 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:18:07.666541 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:18:07.666548 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:18:07.667697 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:18:07.667701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:18:07.668503 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:18:07.668515 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:18:07.669114 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:18:07.669118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:18:07.669202 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:07.669618 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:18:07.669633 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:18:07.670936 (+ 1303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:18:07.672902 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:18:07.672905 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:18:07.672907 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:18:07.673140 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:18:07.673456 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:18:07.673459 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:18:07.673461 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:18:07.673546 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:18:07.673966 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:18:07.673969 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:18:07.674063 (+ 94us) client_negotiation.cc:769] Sending connection context
0924 04:18:07.674138 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 04:18:07.674262 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 04:18:07.674484 22821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:07.663392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37660 (local address 127.0.98.61:42621)
0924 04:18:07.663501 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:07.663504 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:07.663536 (+ 32us) server_negotiation.cc:407] Connection header received
0924 04:18:07.663568 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:07.663571 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:07.663608 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:07.663678 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:07.664420 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:07.665029 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:07.665924 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:07.666282 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:07.667359 (+ 1077us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:07.670638 (+ 3279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:18:07.670641 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:18:07.670644 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:18:07.670667 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:18:07.672334 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:07.672808 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:07.672811 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:07.672813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:07.672888 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:07.673180 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:07.673182 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:07.673184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:07.673391 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:18:07.674154 (+ 763us) server_negotiation.cc:1035] Waiting for connection context
0924 04:18:07.674232 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 04:18:07.674349 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 04:18:07.675020 22897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:07.664999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56846 (local address 127.0.98.60:36213)
0924 04:18:07.666062 (+ 1063us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:07.666069 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:07.666080 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:18:07.666109 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:07.666111 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:07.666147 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:07.666650 (+ 503us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:07.666777 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:07.667547 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:07.668642 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:07.668987 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:07.669393 (+ 406us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:07.671073 (+ 1680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:18:07.671077 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:18:07.671079 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:18:07.671102 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:18:07.672781 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:07.673273 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:07.673276 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:07.673278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:07.673356 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:07.673656 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:07.673658 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:07.673660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:07.673857 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:18:07.674676 (+ 819us) server_negotiation.cc:1035] Waiting for connection context
0924 04:18:07.674761 (+ 85us) server_negotiation.cc:299] Negotiation successful
0924 04:18:07.674867 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":988,"thread_start_us":128,"threads_started":1}
I20260924 04:18:07.675772 22891 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a09394d590524a0dba6ee7f6c6ce475d: Neither blocks nor log segments found. Creating new log.
I20260924 04:18:07.676019 22816 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0584bc9bb958458cb5b9200413d875b1: Neither blocks nor log segments found. Creating new log.
I20260924 04:18:07.676630 22891 log.cc:824] T 00000000000000000000000000000000 P a09394d590524a0dba6ee7f6c6ce475d: Log is configured to *not* fsync() on all Append() calls
I20260924 04:18:07.676803 22816 log.cc:824] T 00000000000000000000000000000000 P 0584bc9bb958458cb5b9200413d875b1: Log is configured to *not* fsync() on all Append() calls
I20260924 04:18:07.677781 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:18:07.677820 22741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb5bcf4217704def817f3d9dd941a6ea: Bootstrap starting.
I20260924 04:18:07.679527 22816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0584bc9bb958458cb5b9200413d875b1: No bootstrap required, opened a new log
I20260924 04:18:07.679704 22891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a09394d590524a0dba6ee7f6c6ce475d: No bootstrap required, opened a new log
I20260924 04:18:07.678995 22713 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "eb5bcf4217704def817f3d9dd941a6ea"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45536
I20260924 04:18:07.680400 22741 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb5bcf4217704def817f3d9dd941a6ea: Neither blocks nor log segments found. Creating new log.
I20260924 04:18:07.681196 22741 log.cc:824] T 00000000000000000000000000000000 P eb5bcf4217704def817f3d9dd941a6ea: Log is configured to *not* fsync() on all Append() calls
I20260924 04:18:07.683341 22741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb5bcf4217704def817f3d9dd941a6ea: No bootstrap required, opened a new log
I20260924 04:18:07.683851 22816 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0584bc9bb958458cb5b9200413d875b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb5bcf4217704def817f3d9dd941a6ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38743 } } peers { permanent_uuid: "0584bc9bb958458cb5b9200413d875b1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42621 } } peers { permanent_uuid: "a09394d590524a0dba6ee7f6c6ce475d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36213 } }
I20260924 04:18:07.684075 22816 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0584bc9bb958458cb5b9200413d875b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0584bc9bb958458cb5b9200413d875b1, State: Initialized, Role: FOLLOWER
I20260924 04:18:07.684018 22891 raft_consensus.cc:348] T 00000000000000000000000000000000 P a09394d590524a0dba6ee7f6c6ce475d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb5bcf4217704def817f3d9dd941a6ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38743 } } peers { permanent_uuid: "0584bc9bb958458cb5b9200413d875b1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42621 } } peers { permanent_uuid: "a09394d590524a0dba6ee7f6c6ce475d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36213 } }
I20260924 04:18:07.684266 22891 raft_consensus.cc:729] T 00000000000000000000000000000000 P a09394d590524a0dba6ee7f6c6ce475d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a09394d590524a0dba6ee7f6c6ce475d, State: Initialized, Role: FOLLOWER
I20260924 04:18:07.684391 22816 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0584bc9bb958458cb5b9200413d875b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb5bcf4217704def817f3d9dd941a6ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38743 } } peers { permanent_uuid: "0584bc9bb958458cb5b9200413d875b1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42621 } } peers { permanent_uuid: "a09394d590524a0dba6ee7f6c6ce475d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36213 } }
I20260924 04:18:07.684567 22891 consensus_queue.cc:260] T 00000000000000000000000000000000 P a09394d590524a0dba6ee7f6c6ce475d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb5bcf4217704def817f3d9dd941a6ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38743 } } peers { permanent_uuid: "0584bc9bb958458cb5b9200413d875b1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42621 } } peers { permanent_uuid: "a09394d590524a0dba6ee7f6c6ce475d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36213 } }
I20260924 04:18:07.685123 22902 sys_catalog.cc:455] T 00000000000000000000000000000000 P a09394d590524a0dba6ee7f6c6ce475d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb5bcf4217704def817f3d9dd941a6ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38743 } } peers { permanent_uuid: "0584bc9bb958458cb5b9200413d875b1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42621 } } peers { permanent_uuid: "a09394d590524a0dba6ee7f6c6ce475d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36213 } } }
I20260924 04:18:07.685422 22902 sys_catalog.cc:458] T 00000000000000000000000000000000 P a09394d590524a0dba6ee7f6c6ce475d [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:18:07.685738 22816 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0584bc9bb958458cb5b9200413d875b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:18:07.685784 22891 sys_catalog.cc:564] T 00000000000000000000000000000000 P a09394d590524a0dba6ee7f6c6ce475d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:18:07.686128 22741 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb5bcf4217704def817f3d9dd941a6ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb5bcf4217704def817f3d9dd941a6ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38743 } } peers { permanent_uuid: "0584bc9bb958458cb5b9200413d875b1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42621 } } peers { permanent_uuid: "a09394d590524a0dba6ee7f6c6ce475d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36213 } }
I20260924 04:18:07.686333 22741 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb5bcf4217704def817f3d9dd941a6ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb5bcf4217704def817f3d9dd941a6ea, State: Initialized, Role: FOLLOWER
I20260924 04:18:07.686650 22741 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb5bcf4217704def817f3d9dd941a6ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb5bcf4217704def817f3d9dd941a6ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38743 } } peers { permanent_uuid: "0584bc9bb958458cb5b9200413d875b1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42621 } } peers { permanent_uuid: "a09394d590524a0dba6ee7f6c6ce475d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36213 } }
I20260924 04:18:07.687191 22913 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb5bcf4217704def817f3d9dd941a6ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb5bcf4217704def817f3d9dd941a6ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38743 } } peers { permanent_uuid: "0584bc9bb958458cb5b9200413d875b1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42621 } } peers { permanent_uuid: "a09394d590524a0dba6ee7f6c6ce475d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36213 } } }
I20260924 04:18:07.687790 22913 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb5bcf4217704def817f3d9dd941a6ea [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:18:07.688241 22741 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb5bcf4217704def817f3d9dd941a6ea [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:18:07.689139 22713 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb5bcf4217704def817f3d9dd941a6ea [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 04:18:07.689245 22713 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb5bcf4217704def817f3d9dd941a6ea [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:18:07.690454 22713 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb5bcf4217704def817f3d9dd941a6ea [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb5bcf4217704def817f3d9dd941a6ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38743 } } peers { permanent_uuid: "0584bc9bb958458cb5b9200413d875b1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42621 } } peers { permanent_uuid: "a09394d590524a0dba6ee7f6c6ce475d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36213 } }
W20260924 04:18:07.691136 22924 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a09394d590524a0dba6ee7f6c6ce475d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:18:07.691200 22924 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:18:07.691419 22713 leader_election.cc:288] T 00000000000000000000000000000000 P eb5bcf4217704def817f3d9dd941a6ea [CANDIDATE]: Term 1 election: Requested vote from peers 0584bc9bb958458cb5b9200413d875b1 (127.0.98.61:42621), a09394d590524a0dba6ee7f6c6ce475d (127.0.98.60:36213)
I20260924 04:18:07.692205 22790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eb5bcf4217704def817f3d9dd941a6ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0584bc9bb958458cb5b9200413d875b1"
I20260924 04:18:07.692205 22864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eb5bcf4217704def817f3d9dd941a6ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a09394d590524a0dba6ee7f6c6ce475d"
I20260924 04:18:07.689388 22822 master_runner.cc:186] Error getting master registration for 127.0.98.62:38743: OK, instance_id { permanent_uuid: "eb5bcf4217704def817f3d9dd941a6ea" instance_seqno: 1790223487199581 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 04:18:07.692416 22864 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a09394d590524a0dba6ee7f6c6ce475d [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:18:07.692416 22790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0584bc9bb958458cb5b9200413d875b1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:18:07.692528 22822 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 04:18:07.693622 22790 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0584bc9bb958458cb5b9200413d875b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb5bcf4217704def817f3d9dd941a6ea in term 1.
I20260924 04:18:07.693622 22864 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a09394d590524a0dba6ee7f6c6ce475d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb5bcf4217704def817f3d9dd941a6ea in term 1.
I20260924 04:18:07.694033 22689 leader_election.cc:302] T 00000000000000000000000000000000 P eb5bcf4217704def817f3d9dd941a6ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a09394d590524a0dba6ee7f6c6ce475d, eb5bcf4217704def817f3d9dd941a6ea; no voters:
W20260924 04:18:07.696386 22694 tablet.cc:2369] T 00000000000000000000000000000000 P eb5bcf4217704def817f3d9dd941a6ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:18:07.693869 22748 master_runner.cc:186] Error getting master registration for 127.0.98.62:38743: OK, instance_id { permanent_uuid: "eb5bcf4217704def817f3d9dd941a6ea" instance_seqno: 1790223487199581 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 04:18:07.697345 22748 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 04:18:07.697574 22913 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb5bcf4217704def817f3d9dd941a6ea [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:18:07.697784 22913 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb5bcf4217704def817f3d9dd941a6ea [term 1 LEADER]: Becoming Leader. State: Replica: eb5bcf4217704def817f3d9dd941a6ea, State: Running, Role: LEADER
I20260924 04:18:07.698294 22913 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb5bcf4217704def817f3d9dd941a6ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb5bcf4217704def817f3d9dd941a6ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38743 } } peers { permanent_uuid: "0584bc9bb958458cb5b9200413d875b1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42621 } } peers { permanent_uuid: "a09394d590524a0dba6ee7f6c6ce475d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36213 } }
I20260924 04:18:07.698791 22901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0584bc9bb958458cb5b9200413d875b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb5bcf4217704def817f3d9dd941a6ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38743 } } peers { permanent_uuid: "0584bc9bb958458cb5b9200413d875b1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42621 } } peers { permanent_uuid: "a09394d590524a0dba6ee7f6c6ce475d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36213 } } }
I20260924 04:18:07.698933 22901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0584bc9bb958458cb5b9200413d875b1 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:18:07.700630 22944 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb5bcf4217704def817f3d9dd941a6ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb5bcf4217704def817f3d9dd941a6ea. Latest consensus state: current_term: 1 leader_uuid: "eb5bcf4217704def817f3d9dd941a6ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb5bcf4217704def817f3d9dd941a6ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38743 } } peers { permanent_uuid: "0584bc9bb958458cb5b9200413d875b1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42621 } } peers { permanent_uuid: "a09394d590524a0dba6ee7f6c6ce475d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36213 } } }
I20260924 04:18:07.700825 22944 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb5bcf4217704def817f3d9dd941a6ea [sys.catalog]: This master's current role is: LEADER
I20260924 04:18:07.702476 22947 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:18:07.704869 22821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:07.693924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37672 (local address 127.0.98.61:42621)
0924 04:18:07.694092 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:07.694097 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:07.694935 (+ 838us) server_negotiation.cc:407] Connection header received
0924 04:18:07.695044 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:07.695048 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:07.695100 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:07.695184 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:07.696040 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:07.696815 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:07.697947 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:07.698263 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:07.698338 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:07.700281 (+ 1943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:18:07.700286 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:18:07.700288 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:18:07.700314 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:18:07.701976 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:07.702496 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:07.702500 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:07.702502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:07.702590 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:07.702896 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:07.702900 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:07.702902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:07.703124 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:18:07.703272 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 04:18:07.704607 (+ 1335us) server_negotiation.cc:299] Negotiation successful
0924 04:18:07.704726 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
W20260924 04:18:07.706243 22925 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0584bc9bb958458cb5b9200413d875b1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:18:07.706316 22925 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:18:07.707994 22947 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:18:07.715276 22897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:07.694338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56848 (local address 127.0.98.60:36213)
0924 04:18:07.694439 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:07.694443 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:07.706393 (+ 11950us) server_negotiation.cc:407] Connection header received
0924 04:18:07.706567 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:07.706571 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:07.706621 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:07.706671 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:07.707610 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:07.708427 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:07.709549 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:07.709916 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:07.709993 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:07.711696 (+ 1703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:18:07.711701 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:18:07.711703 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:18:07.711733 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:18:07.713357 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:07.713845 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:07.713849 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:07.713851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:07.713939 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:07.714251 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:07.714254 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:07.714256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:07.714473 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:18:07.714600 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 04:18:07.714942 (+ 342us) server_negotiation.cc:299] Negotiation successful
0924 04:18:07.715068 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 04:18:07.719033 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:07.694339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45552 (local address 127.0.98.62:38743)
0924 04:18:07.694437 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:07.694441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:07.706581 (+ 12140us) server_negotiation.cc:407] Connection header received
0924 04:18:07.706810 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:07.706813 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:07.706857 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:07.706903 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:07.711724 (+ 4821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:07.712498 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:07.713623 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:07.713948 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:07.714021 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:07.715657 (+ 1636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:18:07.715661 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:18:07.715664 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:18:07.715691 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:18:07.717241 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:07.717759 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:07.717762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:07.717764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:07.717856 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:07.718133 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:07.718136 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:07.718138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:07.718365 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:18:07.718501 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 04:18:07.718744 (+ 243us) server_negotiation.cc:299] Negotiation successful
0924 04:18:07.718864 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 04:18:07.738137 22864 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a09394d590524a0dba6ee7f6c6ce475d [term 1 FOLLOWER]: Refusing update from remote peer eb5bcf4217704def817f3d9dd941a6ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:18:07.738945 22944 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eb5bcf4217704def817f3d9dd941a6ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "a09394d590524a0dba6ee7f6c6ce475d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:18:07.745635 22902 sys_catalog.cc:455] T 00000000000000000000000000000000 P a09394d590524a0dba6ee7f6c6ce475d [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb5bcf4217704def817f3d9dd941a6ea. Latest consensus state: current_term: 1 leader_uuid: "eb5bcf4217704def817f3d9dd941a6ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb5bcf4217704def817f3d9dd941a6ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38743 } } peers { permanent_uuid: "0584bc9bb958458cb5b9200413d875b1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42621 } } peers { permanent_uuid: "a09394d590524a0dba6ee7f6c6ce475d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36213 } } }
I20260924 04:18:07.745766 22902 sys_catalog.cc:458] T 00000000000000000000000000000000 P a09394d590524a0dba6ee7f6c6ce475d [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:18:07.747541 22790 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0584bc9bb958458cb5b9200413d875b1 [term 1 FOLLOWER]: Refusing update from remote peer eb5bcf4217704def817f3d9dd941a6ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:18:07.748164 22944 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eb5bcf4217704def817f3d9dd941a6ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0584bc9bb958458cb5b9200413d875b1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:18:07.752401 22901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0584bc9bb958458cb5b9200413d875b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb5bcf4217704def817f3d9dd941a6ea. Latest consensus state: current_term: 1 leader_uuid: "eb5bcf4217704def817f3d9dd941a6ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb5bcf4217704def817f3d9dd941a6ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38743 } } peers { permanent_uuid: "0584bc9bb958458cb5b9200413d875b1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42621 } } peers { permanent_uuid: "a09394d590524a0dba6ee7f6c6ce475d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36213 } } }
I20260924 04:18:07.752537 22901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0584bc9bb958458cb5b9200413d875b1 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:18:07.755903 22913 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb5bcf4217704def817f3d9dd941a6ea [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eb5bcf4217704def817f3d9dd941a6ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb5bcf4217704def817f3d9dd941a6ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38743 } } peers { permanent_uuid: "0584bc9bb958458cb5b9200413d875b1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42621 } } peers { permanent_uuid: "a09394d590524a0dba6ee7f6c6ce475d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36213 } } }
I20260924 04:18:07.756033 22913 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb5bcf4217704def817f3d9dd941a6ea [sys.catalog]: This master's current role is: LEADER
I20260924 04:18:07.758576 22901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0584bc9bb958458cb5b9200413d875b1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eb5bcf4217704def817f3d9dd941a6ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb5bcf4217704def817f3d9dd941a6ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38743 } } peers { permanent_uuid: "0584bc9bb958458cb5b9200413d875b1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42621 } } peers { permanent_uuid: "a09394d590524a0dba6ee7f6c6ce475d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36213 } } }
I20260924 04:18:07.758685 22901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0584bc9bb958458cb5b9200413d875b1 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:18:07.759493 22950 mvcc.cc:205] Tried to move back new op lower bound from 7332755405755420672 to 7332755405616771072. Current Snapshot: MvccSnapshot[applied={T|T < 7332755405755420672}]
I20260924 04:18:07.762001 22953 mvcc.cc:205] Tried to move back new op lower bound from 7332755405755420672 to 7332755405616771072. Current Snapshot: MvccSnapshot[applied={T|T < 7332755405755420672}]
I20260924 04:18:07.762825 22944 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb5bcf4217704def817f3d9dd941a6ea [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eb5bcf4217704def817f3d9dd941a6ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb5bcf4217704def817f3d9dd941a6ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38743 } } peers { permanent_uuid: "0584bc9bb958458cb5b9200413d875b1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42621 } } peers { permanent_uuid: "a09394d590524a0dba6ee7f6c6ce475d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36213 } } }
I20260924 04:18:07.762948 22944 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb5bcf4217704def817f3d9dd941a6ea [sys.catalog]: This master's current role is: LEADER
I20260924 04:18:07.764015 22902 sys_catalog.cc:455] T 00000000000000000000000000000000 P a09394d590524a0dba6ee7f6c6ce475d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eb5bcf4217704def817f3d9dd941a6ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb5bcf4217704def817f3d9dd941a6ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38743 } } peers { permanent_uuid: "0584bc9bb958458cb5b9200413d875b1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42621 } } peers { permanent_uuid: "a09394d590524a0dba6ee7f6c6ce475d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36213 } } }
I20260924 04:18:07.764132 22902 sys_catalog.cc:458] T 00000000000000000000000000000000 P a09394d590524a0dba6ee7f6c6ce475d [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:18:07.766642 22947 catalog_manager.cc:1347] Generated new cluster ID: 978fca4b79794eedaf8660fc7e3fa5e4
I20260924 04:18:07.766721 22947 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:18:07.769506 22951 mvcc.cc:205] Tried to move back new op lower bound from 7332755405755420672 to 7332755405616771072. Current Snapshot: MvccSnapshot[applied={T|T < 7332755405755420672}]
I20260924 04:18:07.786954 22947 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:18:07.788372 22947 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:18:07.809163 22947 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb5bcf4217704def817f3d9dd941a6ea: Generated new TSK 0
I20260924 04:18:07.810113 22947 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 04:18:07.842422 22744 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:18:07.839375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45554 (local address 127.0.98.62:38743)
0924 04:18:07.839503 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:07.839507 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:07.841137 (+ 1630us) server_negotiation.cc:407] Connection header received
0924 04:18:07.841572 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:07.841576 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:07.841627 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:07.841678 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:07.842283 (+ 605us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45554: BlockingRecv error: recv got EOF from 127.0.0.1:45554 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260924 04:18:07.843778 22821 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:18:07.840237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37676 (local address 127.0.98.61:42621)
0924 04:18:07.840356 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:07.840361 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:07.842924 (+ 2563us) server_negotiation.cc:407] Connection header received
0924 04:18:07.843084 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:07.843087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:07.843133 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:07.843181 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:07.843661 (+ 480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37676: BlockingRecv error: recv got EOF from 127.0.0.1:37676 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260924 04:18:07.845105 22897 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:18:07.840861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56860 (local address 127.0.98.60:36213)
0924 04:18:07.840991 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:07.840995 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:07.844142 (+ 3147us) server_negotiation.cc:407] Connection header received
0924 04:18:07.844397 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:07.844401 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:07.844449 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:07.844498 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:07.844994 (+ 496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56860: BlockingRecv error: recv got EOF from 127.0.0.1:56860 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260924 04:18:07.849006 22821 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:18:07.847460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37680 (local address 127.0.98.61:42621)
0924 04:18:07.847553 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:07.847557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:07.848000 (+ 443us) server_negotiation.cc:407] Connection header received
0924 04:18:07.848307 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:07.848310 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:07.848385 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:07.848436 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:07.848913 (+ 477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37680: BlockingRecv error: recv got EOF from 127.0.0.1:37680 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260924 04:18:07.853787 22821 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:18:07.852449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37696 (local address 127.0.98.61:42621)
0924 04:18:07.852546 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:07.852550 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:07.852976 (+ 426us) server_negotiation.cc:407] Connection header received
0924 04:18:07.853142 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:07.853145 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:07.853193 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:07.853240 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:07.853693 (+ 453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37696: BlockingRecv error: recv got EOF from 127.0.0.1:37696 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260924 04:18:07.859354 22821 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:18:07.857903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37708 (local address 127.0.98.61:42621)
0924 04:18:07.858003 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:07.858007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:07.858466 (+ 459us) server_negotiation.cc:407] Connection header received
0924 04:18:07.858641 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:07.858644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:07.858691 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:07.858741 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:07.859259 (+ 518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37708: BlockingRecv error: recv got EOF from 127.0.0.1:37708 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260924 04:18:07.867255 22821 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:18:07.865627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37714 (local address 127.0.98.61:42621)
0924 04:18:07.865775 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:07.865780 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:07.866294 (+ 514us) server_negotiation.cc:407] Connection header received
0924 04:18:07.866490 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:07.866494 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:07.866546 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:07.866598 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:07.867123 (+ 525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37714: BlockingRecv error: recv got EOF from 127.0.0.1:37714 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260924 04:18:07.879280 22821 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:18:07.877556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37718 (local address 127.0.98.61:42621)
0924 04:18:07.877675 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:07.877679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:07.878311 (+ 632us) server_negotiation.cc:407] Connection header received
0924 04:18:07.878501 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:07.878504 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:07.878554 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:07.878605 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:07.879140 (+ 535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37718: BlockingRecv error: recv got EOF from 127.0.0.1:37718 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260924 04:18:07.898962 22821 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:18:07.897694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37720 (local address 127.0.98.61:42621)
0924 04:18:07.897810 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:07.897815 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:07.898061 (+ 246us) server_negotiation.cc:407] Connection header received
0924 04:18:07.898193 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:07.898197 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:07.898245 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:07.898308 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:07.898844 (+ 536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37720: BlockingRecv error: recv got EOF from 127.0.0.1:37720 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260924 04:18:07.934053 22821 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:18:07.933047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37736 (local address 127.0.98.61:42621)
0924 04:18:07.933175 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:07.933179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:07.933270 (+ 91us) server_negotiation.cc:407] Connection header received
0924 04:18:07.933385 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:07.933387 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:07.933433 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:07.933510 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:07.933938 (+ 428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37736: BlockingRecv error: recv got EOF from 127.0.0.1:37736 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260924 04:18:08.001168 22821 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:18:08.000030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37740 (local address 127.0.98.61:42621)
0924 04:18:08.000175 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:08.000180 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:08.000309 (+ 129us) server_negotiation.cc:407] Connection header received
0924 04:18:08.000487 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:08.000490 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:08.000540 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:08.000603 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:08.001040 (+ 437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37740: BlockingRecv error: recv got EOF from 127.0.0.1:37740 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260924 04:18:08.132359 22821 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:18:08.131178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37752 (local address 127.0.98.61:42621)
0924 04:18:08.131344 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:08.131349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:08.131478 (+ 129us) server_negotiation.cc:407] Connection header received
0924 04:18:08.131624 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:08.131627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:08.131676 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:08.131743 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:08.132205 (+ 462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37752: BlockingRecv error: recv got EOF from 127.0.0.1:37752 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260924 04:18:08.391491 22821 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:18:08.390407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37754 (local address 127.0.98.61:42621)
0924 04:18:08.390519 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:08.390522 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:08.390613 (+ 91us) server_negotiation.cc:407] Connection header received
0924 04:18:08.390725 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:08.390727 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:08.390774 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:08.390851 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:08.391380 (+ 529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37754: BlockingRecv error: recv got EOF from 127.0.0.1:37754 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 04:18:08.692576 22924 catalog_manager.cc:1259] Loaded cluster ID: 978fca4b79794eedaf8660fc7e3fa5e4
I20260924 04:18:08.692662 22924 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a09394d590524a0dba6ee7f6c6ce475d: loading cluster ID for follower catalog manager: success
I20260924 04:18:08.694344 22924 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a09394d590524a0dba6ee7f6c6ce475d: acquiring CA information for follower catalog manager: success
I20260924 04:18:08.695550 22924 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a09394d590524a0dba6ee7f6c6ce475d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 04:18:08.707615 22925 catalog_manager.cc:1259] Loaded cluster ID: 978fca4b79794eedaf8660fc7e3fa5e4
I20260924 04:18:08.707686 22925 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0584bc9bb958458cb5b9200413d875b1: loading cluster ID for follower catalog manager: success
I20260924 04:18:08.709363 22925 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0584bc9bb958458cb5b9200413d875b1: acquiring CA information for follower catalog manager: success
I20260924 04:18:08.710512 22925 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0584bc9bb958458cb5b9200413d875b1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 04:18:08.909094 23024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:08.905330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37768 (local address 127.0.98.61:42621)
0924 04:18:08.905560 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:08.905563 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:08.905672 (+ 109us) server_negotiation.cc:407] Connection header received
0924 04:18:08.905767 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:08.905769 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:08.905902 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:08.906004 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:18:08.906211 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:08.906726 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:08.907697 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:08.907939 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:08.908019 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:08.908340 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:18:08.908771 (+ 431us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:18:08.908877 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 04:18:08.908921 (+ 44us) server_negotiation.cc:299] Negotiation successful
0924 04:18:08.908971 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":66,"threads_started":1}
I20260924 04:18:08.915184 23030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:08.912103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56872 (local address 127.0.98.60:36213)
0924 04:18:08.912317 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:08.912320 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:08.912336 (+ 16us) server_negotiation.cc:407] Connection header received
0924 04:18:08.912413 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:08.912415 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:08.912524 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:08.912616 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:18:08.912773 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:08.913278 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:08.914025 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:08.914227 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:08.914292 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:08.914495 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:18:08.914901 (+ 406us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:18:08.914982 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 04:18:08.915023 (+ 41us) server_negotiation.cc:299] Negotiation successful
0924 04:18:08.915068 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":78,"threads_started":1}
I20260924 04:18:08.922008 23030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:08.918326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56884 (local address 127.0.98.60:36213)
0924 04:18:08.918424 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:08.918428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:08.918502 (+ 74us) server_negotiation.cc:407] Connection header received
0924 04:18:08.918579 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:08.918582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:08.918622 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:08.918695 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:18:08.918934 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:08.919642 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:08.920899 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:08.921228 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:08.921313 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:08.921571 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:18:08.921653 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:18:08.921693 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 04:18:08.921823 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 04:18:08.921872 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 04:18:08.922477 23037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:08.917917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45562 (local address 127.0.98.62:38743)
0924 04:18:08.918167 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:08.918172 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:08.918391 (+ 219us) server_negotiation.cc:407] Connection header received
0924 04:18:08.918624 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:08.918628 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:08.918793 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:08.918882 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:18:08.919219 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:08.919915 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:08.921012 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:08.921361 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:08.921459 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:08.921755 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:18:08.922104 (+ 349us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:18:08.922206 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 04:18:08.922273 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 04:18:08.922332 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":75,"threads_started":1}
I20260924 04:18:08.924556 23024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:08.917967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37774 (local address 127.0.98.61:42621)
0924 04:18:08.918091 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:08.918096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:08.918472 (+ 376us) server_negotiation.cc:407] Connection header received
0924 04:18:08.918675 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:08.918678 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:08.918733 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:08.918799 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:18:08.919047 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:08.919622 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:08.923492 (+ 3870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:08.923805 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:08.923916 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:08.924147 (+ 231us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:18:08.924236 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:18:08.924304 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 04:18:08.924392 (+ 88us) server_negotiation.cc:299] Negotiation successful
0924 04:18:08.924438 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 04:18:08.925807 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 22673
I20260924 04:18:08.930979 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 22748
I20260924 04:18:08.935669 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 22822
2026-09-24T04:18:08Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2971 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:18:08 dist-test-slave-p401 krb5kdc[23041](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:18:08 dist-test-slave-p401 krb5kdc[23041](info): set up 2 sockets
Sep 24 04:18:08 dist-test-slave-p401 krb5kdc[23041](info): commencing operation
krb5kdc: starting...
W20260924 04:18:09.883405 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/krb5kdc/test-user.keytab.
Sep 24 04:18:09 dist-test-slave-p401 krb5kdc[23041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:18:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:18:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:18:10.002089 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41415
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41415,127.0.98.61:45645,127.0.98.60:39319
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:18:10.109441 23054 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 04:18:10.109773 23054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:18:10.109815 23054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:18:10.109876 23054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:18:10.113937 23054 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:18:10.114004 23054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:18:10.114023 23054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:18:10.114038 23054 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:18:10.114053 23054 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:18:10.119336 23054 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41415,127.0.98.61:45645,127.0.98.60:39319
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:18:10.120322 23054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:18:10.121249 23054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:18:10.129464 23059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:18:10.129566 23062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:18:10.129566 23060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:18:10.129509 23054 server_base.cc:1034] running on GCE node
I20260924 04:18:10.130307 23054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:18:10.131454 23054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:18:10.132628 23054 hybrid_clock.cc:648] HybridClock initialized: now 1790223490132614 us; error 32 us; skew 500 ppm
Sep 24 04:18:10 dist-test-slave-p401 krb5kdc[23041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:18:10.140434 23054 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:18:10.141417 23054 webserver.cc:460] Webserver started at http://127.0.98.62:34605/ using document root <none> and password file <none>
I20260924 04:18:10.141968 23054 fs_manager.cc:359] Metadata directory not provided
I20260924 04:18:10.142032 23054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:18:10.142233 23054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:18:10.144302 23054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/minicluster-data/master-0/data/instance:
uuid: "b3405a778eb94bc19b4401e00a44e8f7"
format_stamp: "Formatted at 2026-09-24 04:18:10 on dist-test-slave-p401"
server_key: "49bd8fe536e74566f9e7c868329acf19"
server_key_iv: "29b34b9fa4215bf7a8a53c74a4173133"
server_key_version: "encryptionkey@0"
I20260924 04:18:10.144829 23054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/minicluster-data/master-0/wal/instance:
uuid: "b3405a778eb94bc19b4401e00a44e8f7"
format_stamp: "Formatted at 2026-09-24 04:18:10 on dist-test-slave-p401"
server_key: "49bd8fe536e74566f9e7c868329acf19"
server_key_iv: "29b34b9fa4215bf7a8a53c74a4173133"
server_key_version: "encryptionkey@0"
I20260924 04:18:10.148342 23054 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:18:10.150630 23069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:18:10.151816 23054 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:18:10.151952 23054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/minicluster-data/master-0/wal
uuid: "b3405a778eb94bc19b4401e00a44e8f7"
format_stamp: "Formatted at 2026-09-24 04:18:10 on dist-test-slave-p401"
server_key: "49bd8fe536e74566f9e7c868329acf19"
server_key_iv: "29b34b9fa4215bf7a8a53c74a4173133"
server_key_version: "encryptionkey@0"
I20260924 04:18:10.152047 23054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:18:10.188843 23054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:18:10.192015 23054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:18:10.192183 23054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:18:10.198962 23054 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41415
I20260924 04:18:10.199337 23121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41415 every 8 connection(s)
I20260924 04:18:10.200006 23054 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:18:10.202775 23122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:18:10.206429 23122 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41415 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:18:10.208339 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 23054
I20260924 04:18:10.208448 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/minicluster-data/master-0/wal/instance
I20260924 04:18:10.208698 392 external_mini_cluster.cc:1428] Setting key 6397a5cf1ccd6f4cd3cde24218b0e533
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 24 04:18:10 dist-test-slave-p401 krb5kdc[23041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:18:10.225039 23125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:10.208792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41415 (local address 127.0.0.1:44648)
0924 04:18:10.209415 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0924 04:18:10.209464 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 04:18:10.210425 (+ 961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:18:10.211127 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:18:10.211132 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:18:10.211669 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:18:10.212146 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:18:10.212167 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:18:10.213413 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:18:10.213416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:18:10.214588 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:18:10.214599 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:18:10.215272 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:18:10.215277 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:18:10.215405 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:10.215956 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:18:10.215977 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:18:10.220165 (+ 4188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:18:10.222045 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:18:10.222055 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:18:10.222066 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:18:10.222359 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:18:10.222654 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:18:10.222657 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:18:10.222659 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:18:10.222796 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:18:10.223487 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:18:10.223494 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:18:10.223666 (+ 172us) client_negotiation.cc:769] Sending connection context
0924 04:18:10.223866 (+ 200us) client_negotiation.cc:240] Negotiation successful
0924 04:18:10.224303 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"mutex_wait_us":88,"spinlock_wait_cycles":52992,"thread_start_us":150,"threads_started":1}
I20260924 04:18:10.225036 23124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:10.208792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44648 (local address 127.0.98.62:41415)
0924 04:18:10.209286 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:10.209304 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:10.209570 (+ 266us) server_negotiation.cc:407] Connection header received
0924 04:18:10.210729 (+ 1159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:10.210733 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:10.210926 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:10.211077 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:10.212324 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:10.213275 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:10.214717 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:10.215087 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:10.215202 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:10.220358 (+ 5156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:18:10.220366 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:18:10.220380 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:18:10.220407 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:18:10.221945 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:10.222464 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:10.222469 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:10.222470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:10.222572 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:10.222890 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:10.222893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:10.222895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:10.223378 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:18:10.223481 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 04:18:10.223956 (+ 475us) server_negotiation.cc:299] Negotiation successful
0924 04:18:10.224202 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":323,"thread_start_us":146,"threads_started":1}
I20260924 04:18:10.227433 23122 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45645 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:18:10.228139 23125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:10.227881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45645 (local address 127.0.0.1:35740)
0924 04:18:10.228007 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 04:18:10.228078 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 04:18:10.228368 23070 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45645: connect: Connection refused (error 111)
I20260924 04:18:10.229161 23125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:10.228967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45645 (local address 127.0.0.1:35756)
0924 04:18:10.229068 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0924 04:18:10.229111 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 04:18:10.229279 23122 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:45645: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45645: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260924 04:18:10.234221 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45645
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41415,127.0.98.61:45645,127.0.98.60:39319
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:18:10.278535 23122 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45645 } attempt: 1
I20260924 04:18:10.279448 23125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:10.279123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45645 (local address 127.0.0.1:35758)
0924 04:18:10.279306 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 04:18:10.279372 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260924 04:18:10.280093 23125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:10.279894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45645 (local address 127.0.0.1:35768)
0924 04:18:10.279994 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0924 04:18:10.280036 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260924 04:18:10.280200 23122 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:45645: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45645: connect: Connection refused (error 111)
W20260924 04:18:10.344754 23129 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 04:18:10.345019 23129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:18:10.345052 23129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:18:10.345112 23129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 04:18:10.348348 23122 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45645 } attempt: 2
I20260924 04:18:10.349118 23125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:10.348844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45645 (local address 127.0.0.1:35784)
0924 04:18:10.348982 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 04:18:10.349042 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 04:18:10.349138 23129 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:18:10.349184 23129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:18:10.349211 23129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:18:10.349293 23129 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:18:10.349325 23129 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:18:10.349900 23125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:10.349690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45645 (local address 127.0.0.1:35786)
0924 04:18:10.349787 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0924 04:18:10.349847 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260924 04:18:10.350013 23122 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:45645: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45645: connect: Connection refused (error 111)
I20260924 04:18:10.354593 23129 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41415,127.0.98.61:45645,127.0.98.60:39319
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:18:10.355631 23129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:18:10.356595 23129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:18:10.364367 23137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:18:10.364413 23135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:18:10.364449 23134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:18:10.364615 23129 server_base.cc:1034] running on GCE node
I20260924 04:18:10.365213 23129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:18:10.366246 23129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:18:10.367401 23129 hybrid_clock.cc:648] HybridClock initialized: now 1790223490367381 us; error 31 us; skew 500 ppm
Sep 24 04:18:10 dist-test-slave-p401 krb5kdc[23041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:18:10.369921 23129 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260924 04:18:10.370769 23129 webserver.cc:460] Webserver started at http://127.0.98.61:41269/ using document root <none> and password file <none>
I20260924 04:18:10.371382 23129 fs_manager.cc:359] Metadata directory not provided
I20260924 04:18:10.371459 23129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:18:10.371657 23129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:18:10.373654 23129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/minicluster-data/master-1/data/instance:
uuid: "4463e65521a34a19a456696ada697046"
format_stamp: "Formatted at 2026-09-24 04:18:10 on dist-test-slave-p401"
server_key: "c63f35201da03d0461e1005f99ed2160"
server_key_iv: "2041c60c943753a4cb34ba41ef87c91c"
server_key_version: "encryptionkey@0"
I20260924 04:18:10.374197 23129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/minicluster-data/master-1/wal/instance:
uuid: "4463e65521a34a19a456696ada697046"
format_stamp: "Formatted at 2026-09-24 04:18:10 on dist-test-slave-p401"
server_key: "c63f35201da03d0461e1005f99ed2160"
server_key_iv: "2041c60c943753a4cb34ba41ef87c91c"
server_key_version: "encryptionkey@0"
I20260924 04:18:10.377911 23129 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 04:18:10.380414 23144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:18:10.381593 23129 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:18:10.381713 23129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/minicluster-data/master-1/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/minicluster-data/master-1/wal
uuid: "4463e65521a34a19a456696ada697046"
format_stamp: "Formatted at 2026-09-24 04:18:10 on dist-test-slave-p401"
server_key: "c63f35201da03d0461e1005f99ed2160"
server_key_iv: "2041c60c943753a4cb34ba41ef87c91c"
server_key_version: "encryptionkey@0"
I20260924 04:18:10.381810 23129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:18:10.401602 23129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:18:10.404680 23129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:18:10.404868 23129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:18:10.411687 23129 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:45645
I20260924 04:18:10.411993 23196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:45645 every 8 connection(s)
I20260924 04:18:10.412691 23129 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/minicluster-data/master-1/data/info.pb
I20260924 04:18:10.415470 23197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:18:10.419082 23197 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41415 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:18:10.420280 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 23129
I20260924 04:18:10.420436 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/minicluster-data/master-1/wal/instance
I20260924 04:18:10.420689 392 external_mini_cluster.cc:1428] Setting key ec151f0a378a172e4bcb2a75b3c70b4a
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 24 04:18:10 dist-test-slave-p401 krb5kdc[23041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:18:10.434371 23124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:10.420970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44660 (local address 127.0.98.62:41415)
0924 04:18:10.421118 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:10.421123 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:10.421965 (+ 842us) server_negotiation.cc:407] Connection header received
0924 04:18:10.422957 (+ 992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:10.422960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:10.423016 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:10.423111 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:10.424266 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:10.425075 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:10.426171 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:10.426522 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:10.426634 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:10.430939 (+ 4305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:18:10.430942 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:18:10.430944 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:18:10.430969 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:18:10.432456 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:10.432926 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:10.432929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:10.432931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:10.433006 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:10.433313 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:10.433316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:10.433318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:10.433494 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:18:10.433577 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 04:18:10.434084 (+ 507us) server_negotiation.cc:299] Negotiation successful
0924 04:18:10.434205 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 04:18:10.435145 23199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:10.421339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41415 (local address 127.0.0.1:44660)
0924 04:18:10.421809 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0924 04:18:10.421862 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 04:18:10.422752 (+ 890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:18:10.423285 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:18:10.423293 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:18:10.423660 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:18:10.424123 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:18:10.424136 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:18:10.425193 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:18:10.425196 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:18:10.426054 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:18:10.426062 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:18:10.426674 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:18:10.426677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:18:10.426815 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:10.427387 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:18:10.427403 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:18:10.430774 (+ 3371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:18:10.432565 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:18:10.432571 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:18:10.432587 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:18:10.432834 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:18:10.433087 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:18:10.433090 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:18:10.433092 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:18:10.433228 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:18:10.433588 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:18:10.433595 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:18:10.433847 (+ 252us) client_negotiation.cc:769] Sending connection context
0924 04:18:10.434009 (+ 162us) client_negotiation.cc:240] Negotiation successful
0924 04:18:10.434340 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":111,"threads_started":1}
I20260924 04:18:10.436607 23197 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45645 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:18:10.441148 23122 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45645 } attempt: 3
Sep 24 04:18:10 dist-test-slave-p401 krb5kdc[23041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 04:18:10.445411 23199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:10.437101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45645 (local address 127.0.0.1:35792)
0924 04:18:10.437203 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0924 04:18:10.437217 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:18:10.437291 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:18:10.437823 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:18:10.437826 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:18:10.437964 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:18:10.438052 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:18:10.438058 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:18:10.439067 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:18:10.439070 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:18:10.439647 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:18:10.439655 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:18:10.440221 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:18:10.440224 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:18:10.440296 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:10.440718 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:18:10.440729 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:18:10.442034 (+ 1305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:18:10.443932 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:18:10.443935 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:18:10.443937 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:18:10.444133 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:18:10.444409 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:18:10.444412 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:18:10.444414 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:18:10.444507 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:18:10.444945 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:18:10.444948 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:18:10.445031 (+ 83us) client_negotiation.cc:769] Sending connection context
0924 04:18:10.445115 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 04:18:10.445234 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 04:18:10.445673 23202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:10.437179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35792 (local address 127.0.98.61:45645)
0924 04:18:10.437468 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:10.437477 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:10.437496 (+ 19us) server_negotiation.cc:407] Connection header received
0924 04:18:10.437529 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:10.437533 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:10.437686 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:10.437843 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:10.438139 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:10.438963 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:10.439762 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:10.440126 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:10.440208 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:10.442145 (+ 1937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:18:10.442152 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:18:10.442163 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:18:10.442182 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:18:10.443822 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:10.444224 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:10.444230 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:10.444231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:10.444306 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:10.444578 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:10.444581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:10.444583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:10.444820 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:18:10.444948 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 04:18:10.445203 (+ 255us) server_negotiation.cc:299] Negotiation successful
0924 04:18:10.445328 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":120,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Sep 24 04:18:10 dist-test-slave-p401 krb5kdc[23041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 04:18:10.446981 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6HV92s/build/debug/bin/kudu
/tmp/dist-test-task6HV92s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39319
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41415,127.0.98.61:45645,127.0.98.60:39319
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:18:10.447786 23197 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39319 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:18:10.448470 23199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:10.448208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39319 (local address 127.0.0.1:57738)
0924 04:18:10.448323 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 04:18:10.448406 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260924 04:18:10.448724 23148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39319: connect: Connection refused (error 111)
I20260924 04:18:10.449647 23199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:10.449474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39319 (local address 127.0.0.1:57748)
0924 04:18:10.449556 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0924 04:18:10.449596 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 04:18:10.449985 23197 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39319: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39319: connect: Connection refused (error 111)
I20260924 04:18:10.450987 23203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:10.441652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35794 (local address 127.0.98.61:45645)
0924 04:18:10.441970 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:10.441977 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:10.442010 (+ 33us) server_negotiation.cc:407] Connection header received
0924 04:18:10.442037 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:10.442038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:10.442071 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:10.442141 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:10.442590 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:10.443113 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:10.443963 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:10.444179 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:10.444261 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:10.447350 (+ 3089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:18:10.447353 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:18:10.447355 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:18:10.447375 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:18:10.449275 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:10.449903 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:10.449905 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:10.449906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:10.449983 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:10.450255 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:10.450257 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:10.450258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:10.450448 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:18:10.450521 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 04:18:10.450787 (+ 266us) server_negotiation.cc:299] Negotiation successful
0924 04:18:10.450878 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":110,"threads_started":1}
I20260924 04:18:10.451031 23125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:10.441571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45645 (local address 127.0.0.1:35794)
0924 04:18:10.441676 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0924 04:18:10.441690 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:18:10.441764 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:18:10.442188 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:18:10.442191 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:18:10.442346 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:18:10.442461 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:18:10.442468 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:18:10.443250 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:18:10.443254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:18:10.443829 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:18:10.443839 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:18:10.444276 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:18:10.444280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:18:10.444391 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:10.444773 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:18:10.444787 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:18:10.447179 (+ 2392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:18:10.449586 (+ 2407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:18:10.449589 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:18:10.449592 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:18:10.449822 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:18:10.450080 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:18:10.450083 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:18:10.450085 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:18:10.450179 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:18:10.450538 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:18:10.450541 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:18:10.450640 (+ 99us) client_negotiation.cc:769] Sending connection context
0924 04:18:10.450740 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 04:18:10.450882 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 04:18:10.451851 23122 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39319 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:18:10.452476 23125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:10.452222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39319 (local address 127.0.0.1:57764)
0924 04:18:10.452348 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 04:18:10.452404 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 04:18:10.453104 23125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:10.452862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39319 (local address 127.0.0.1:57780)
0924 04:18:10.452997 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 04:18:10.453046 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260924 04:18:10.453234 23122 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39319: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39319: connect: Connection refused (error 111)
I20260924 04:18:10.484385 23122 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39319 } attempt: 1
I20260924 04:18:10.485319 23125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:10.485023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39319 (local address 127.0.0.1:57790)
0924 04:18:10.485176 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 04:18:10.485238 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 04:18:10.486088 23125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:10.485817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39319 (local address 127.0.0.1:57806)
0924 04:18:10.485986 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 04:18:10.486032 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260924 04:18:10.486243 23122 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39319: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39319: connect: Connection refused (error 111)
I20260924 04:18:10.499276 23197 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39319 } attempt: 1
I20260924 04:18:10.500083 23199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:10.499797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39319 (local address 127.0.0.1:57818)
0924 04:18:10.499945 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 04:18:10.500007 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 04:18:10.500747 23199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:10.500540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39319 (local address 127.0.0.1:57824)
0924 04:18:10.500643 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0924 04:18:10.500688 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260924 04:18:10.500859 23197 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39319: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39319: connect: Connection refused (error 111)
W20260924 04:18:10.556968 23204 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 04:18:10.557255 23204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:18:10.557297 23204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:18:10.557360 23204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:18:10.561383 23204 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:18:10.561452 23204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:18:10.561480 23204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
I20260924 04:18:10.561403 23122 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39319 } attempt: 2
W20260924 04:18:10.561502 23204 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:18:10.561528 23204 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:18:10.562181 23125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:10.561901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39319 (local address 127.0.0.1:57838)
0924 04:18:10.562042 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 04:18:10.562104 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 04:18:10.562845 23125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:10.562641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39319 (local address 127.0.0.1:57848)
0924 04:18:10.562739 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0924 04:18:10.562782 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 04:18:10.562981 23122 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39319: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39319: connect: Connection refused (error 111)
I20260924 04:18:10.567102 23204 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41415,127.0.98.61:45645,127.0.98.60:39319
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:18:10.568204 23204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:18:10.569010 23197 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39319 } attempt: 2
I20260924 04:18:10.569264 23204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:18:10.569748 23199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:10.569481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39319 (local address 127.0.0.1:57860)
0924 04:18:10.569622 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 04:18:10.569680 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 04:18:10.570396 23199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:10.570193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39319 (local address 127.0.0.1:57870)
0924 04:18:10.570288 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0924 04:18:10.570341 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 04:18:10.570494 23197 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39319: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39319: connect: Connection refused (error 111)
W20260924 04:18:10.577739 23211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:18:10.577723 23210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:18:10.577834 23204 server_base.cc:1034] running on GCE node
W20260924 04:18:10.577739 23213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:18:10.578545 23204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:18:10.579696 23204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:18:10.580870 23204 hybrid_clock.cc:648] HybridClock initialized: now 1790223490580844 us; error 35 us; skew 500 ppm
Sep 24 04:18:10 dist-test-slave-p401 krb5kdc[23041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:18:10.583606 23204 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260924 04:18:10.584520 23204 webserver.cc:460] Webserver started at http://127.0.98.60:36795/ using document root <none> and password file <none>
I20260924 04:18:10.585124 23204 fs_manager.cc:359] Metadata directory not provided
I20260924 04:18:10.585194 23204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:18:10.585412 23204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:18:10.587504 23204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/minicluster-data/master-2/data/instance:
uuid: "7ede8d1d8476411a90984466081dc9cd"
format_stamp: "Formatted at 2026-09-24 04:18:10 on dist-test-slave-p401"
server_key: "458af26201face9c6f6251eedab8c83f"
server_key_iv: "ac3237d19eb8957097139d3c34cdf3f3"
server_key_version: "encryptionkey@0"
I20260924 04:18:10.588032 23204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/minicluster-data/master-2/wal/instance:
uuid: "7ede8d1d8476411a90984466081dc9cd"
format_stamp: "Formatted at 2026-09-24 04:18:10 on dist-test-slave-p401"
server_key: "458af26201face9c6f6251eedab8c83f"
server_key_iv: "ac3237d19eb8957097139d3c34cdf3f3"
server_key_version: "encryptionkey@0"
I20260924 04:18:10.591756 23204 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:18:10.594069 23220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:18:10.595294 23204 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:18:10.595402 23204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/minicluster-data/master-2/data,/tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/minicluster-data/master-2/wal
uuid: "7ede8d1d8476411a90984466081dc9cd"
format_stamp: "Formatted at 2026-09-24 04:18:10 on dist-test-slave-p401"
server_key: "458af26201face9c6f6251eedab8c83f"
server_key_iv: "ac3237d19eb8957097139d3c34cdf3f3"
server_key_version: "encryptionkey@0"
I20260924 04:18:10.595492 23204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:18:10.606469 23204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:18:10.609344 23204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:18:10.609514 23204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:18:10.615918 23204 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:39319
I20260924 04:18:10.616166 23272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:39319 every 8 connection(s)
I20260924 04:18:10.616842 23204 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/minicluster-data/master-2/data/info.pb
I20260924 04:18:10.619503 23273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:18:10.623034 23273 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41415 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:18:10.624701 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6HV92s/build/debug/bin/kudu as pid 23204
I20260924 04:18:10.624821 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6HV92s/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790223216265354-392-0/minicluster-data/master-2/wal/instance
I20260924 04:18:10.625113 392 external_mini_cluster.cc:1428] Setting key 6fa0d8482bd0e4b645487bc4f092e215
Sep 24 04:18:10 dist-test-slave-p401 krb5kdc[23041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 24 04:18:10 dist-test-slave-p401 krb5kdc[23041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 04:18:10.636034 23124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:10.624830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44672 (local address 127.0.98.62:41415)
0924 04:18:10.624975 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:10.624980 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:10.625866 (+ 886us) server_negotiation.cc:407] Connection header received
0924 04:18:10.626975 (+ 1109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:10.626978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:10.627021 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:10.627083 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:10.627964 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:10.628462 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:10.629500 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:10.629771 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:10.629854 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:10.632282 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:18:10.632285 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:18:10.632288 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:18:10.632312 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:18:10.633915 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:10.634496 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:10.634500 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:10.634502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:10.634577 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:10.634899 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:10.634903 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:10.634904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:10.635121 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:18:10.635215 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 04:18:10.635737 (+ 522us) server_negotiation.cc:299] Negotiation successful
0924 04:18:10.635864 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:18:10.636832 23275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:10.625219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41415 (local address 127.0.0.1:44672)
0924 04:18:10.625684 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0924 04:18:10.625735 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 04:18:10.626759 (+ 1024us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:18:10.627146 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:18:10.627151 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:18:10.627581 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:18:10.627821 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:18:10.627845 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:18:10.628559 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:18:10.628562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:18:10.629396 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:18:10.629404 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:18:10.629920 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:18:10.629924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:18:10.630100 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:10.630652 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:18:10.630667 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:18:10.632149 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:18:10.634043 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:18:10.634050 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:18:10.634058 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:18:10.634397 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:18:10.634655 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:18:10.634658 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:18:10.634660 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:18:10.634814 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:18:10.635216 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:18:10.635221 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:18:10.635507 (+ 286us) client_negotiation.cc:769] Sending connection context
0924 04:18:10.635673 (+ 166us) client_negotiation.cc:240] Negotiation successful
0924 04:18:10.635979 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"spinlock_wait_cycles":4992,"thread_start_us":109,"threads_started":1}
I20260924 04:18:10.636831 23277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:10.625728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57884 (local address 127.0.98.60:39319)
0924 04:18:10.626120 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:10.626131 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:10.626252 (+ 121us) server_negotiation.cc:407] Connection header received
0924 04:18:10.626764 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:10.626784 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:10.627024 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:10.627184 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:10.627887 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:10.628864 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:10.629695 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:10.630120 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:10.630211 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:10.632708 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:18:10.632715 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:18:10.632725 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:18:10.632746 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:18:10.634375 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:10.634767 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:10.634772 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:10.634773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:10.634843 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:10.635076 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:10.635078 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:10.635080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:10.635485 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:18:10.635625 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 04:18:10.635858 (+ 233us) server_negotiation.cc:299] Negotiation successful
0924 04:18:10.636037 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":312,"thread_start_us":132,"threads_started":1}
I20260924 04:18:10.637934 23273 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45645 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 04:18:10 dist-test-slave-p401 krb5kdc[23041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 24 04:18:10 dist-test-slave-p401 krb5kdc[23041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:18:10.645432 23275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:10.638388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45645 (local address 127.0.0.1:35798)
0924 04:18:10.638497 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 04:18:10.638510 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:18:10.638576 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:18:10.638801 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:18:10.638803 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:18:10.638960 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:18:10.639073 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:18:10.639080 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:18:10.639838 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:18:10.639841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:18:10.640344 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:18:10.640351 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:18:10.640769 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:18:10.640772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:18:10.640872 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:10.641289 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:18:10.641303 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:18:10.642477 (+ 1174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:18:10.644152 (+ 1675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:18:10.644155 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:18:10.644158 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:18:10.644338 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:18:10.644577 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:18:10.644580 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:18:10.644582 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:18:10.644651 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:18:10.644986 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:18:10.644989 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:18:10.645074 (+ 85us) client_negotiation.cc:769] Sending connection context
0924 04:18:10.645154 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 04:18:10.645274 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 04:18:10.645458 23203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:10.638457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35798 (local address 127.0.98.61:45645)
0924 04:18:10.638570 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:10.638573 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:10.638584 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:18:10.638667 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:10.638670 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:10.638717 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:10.638788 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:10.639172 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:10.639733 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:10.640441 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:10.640679 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:10.640764 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:10.642606 (+ 1842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:18:10.642610 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:18:10.642613 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:18:10.642643 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:18:10.644053 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:10.644422 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:10.644425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:10.644427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:10.644499 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:10.644723 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:10.644726 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:10.644728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:10.644903 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:18:10.644984 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 04:18:10.645210 (+ 226us) server_negotiation.cc:299] Negotiation successful
0924 04:18:10.645329 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:18:10.646353 23273 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39319 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:18:10.646683 23124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:10.639085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44688 (local address 127.0.98.62:41415)
0924 04:18:10.639180 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:10.639184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:10.639195 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:18:10.639302 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:10.639305 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:10.639343 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:10.639406 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:10.639932 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:10.640391 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:10.641104 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:10.641381 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:10.641480 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:10.643741 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:18:10.643744 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:18:10.643746 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:18:10.643766 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:18:10.644958 (+ 1192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:10.645313 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:10.645317 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:10.645318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:10.645413 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:10.645731 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:10.645735 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:10.645736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:10.645928 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:18:10.646117 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0924 04:18:10.646442 (+ 325us) server_negotiation.cc:299] Negotiation successful
0924 04:18:10.646546 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Sep 24 04:18:10 dist-test-slave-p401 krb5kdc[23041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 04:18:10 dist-test-slave-p401 krb5kdc[23041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 04:18:10.653910 23275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:10.646782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39319 (local address 127.0.0.1:57888)
0924 04:18:10.646883 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0924 04:18:10.646895 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:18:10.646958 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:18:10.647167 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:18:10.647170 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:18:10.647355 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:18:10.647438 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:18:10.647444 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:18:10.648368 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:18:10.648370 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:18:10.649017 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:18:10.649026 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:18:10.649482 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:18:10.649486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:18:10.649550 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:10.649931 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:18:10.649941 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:18:10.650992 (+ 1051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:18:10.652577 (+ 1585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:18:10.652581 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:18:10.652583 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:18:10.652772 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:18:10.653030 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:18:10.653033 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:18:10.653035 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:18:10.653108 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:18:10.653455 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:18:10.653458 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:18:10.653549 (+ 91us) client_negotiation.cc:769] Sending connection context
0924 04:18:10.653654 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 04:18:10.653776 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 04:18:10.654040 23277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:10.646899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57888 (local address 127.0.98.60:39319)
0924 04:18:10.646991 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:10.646996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:10.647009 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:18:10.647038 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:10.647040 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:10.647076 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:10.647147 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:10.647526 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:10.648252 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:10.649119 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:10.649387 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:10.649469 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:10.651101 (+ 1632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:18:10.651105 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:18:10.651107 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:18:10.651127 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:18:10.652471 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:10.652870 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:10.652873 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:10.652875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:10.652956 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:10.653183 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:10.653185 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:10.653186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:10.653374 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:18:10.653464 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 04:18:10.653702 (+ 238us) server_negotiation.cc:299] Negotiation successful
0924 04:18:10.653861 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":29}
I20260924 04:18:10.655982 23203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:10.647709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35802 (local address 127.0.98.61:45645)
0924 04:18:10.647809 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:10.647813 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:10.647840 (+ 27us) server_negotiation.cc:407] Connection header received
0924 04:18:10.647948 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:10.647950 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:10.647981 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:10.648046 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:10.648660 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:10.649368 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:10.650237 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:10.650512 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:10.650600 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:10.652986 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:18:10.652988 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:18:10.652990 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:18:10.653014 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:18:10.654552 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:10.654894 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:10.654897 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:10.654899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:10.654968 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:10.655213 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:10.655217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:10.655218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:10.655435 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:18:10.655629 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0924 04:18:10.655740 (+ 111us) server_negotiation.cc:299] Negotiation successful
0924 04:18:10.655847 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 04:18:10.657542 23273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ede8d1d8476411a90984466081dc9cd: Bootstrap starting.
I20260924 04:18:10.658990 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:18:10.660013 23273 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ede8d1d8476411a90984466081dc9cd: Neither blocks nor log segments found. Creating new log.
I20260924 04:18:10.660902 23273 log.cc:824] T 00000000000000000000000000000000 P 7ede8d1d8476411a90984466081dc9cd: Log is configured to *not* fsync() on all Append() calls
I20260924 04:18:10.660106 23096 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "b3405a778eb94bc19b4401e00a44e8f7"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44688
I20260924 04:18:10.661643 23197 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39319 } attempt: 3
I20260924 04:18:10.662086 23122 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39319 } attempt: 3
I20260924 04:18:10.663199 23273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ede8d1d8476411a90984466081dc9cd: No bootstrap required, opened a new log
Sep 24 04:18:10 dist-test-slave-p401 krb5kdc[23041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 04:18:10.667208 23273 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ede8d1d8476411a90984466081dc9cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3405a778eb94bc19b4401e00a44e8f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41415 } } peers { permanent_uuid: "4463e65521a34a19a456696ada697046" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45645 } } peers { permanent_uuid: "7ede8d1d8476411a90984466081dc9cd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39319 } }
I20260924 04:18:10.667508 23273 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ede8d1d8476411a90984466081dc9cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ede8d1d8476411a90984466081dc9cd, State: Initialized, Role: FOLLOWER
Sep 24 04:18:10 dist-test-slave-p401 krb5kdc[23041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 04:18:10.667881 23273 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ede8d1d8476411a90984466081dc9cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3405a778eb94bc19b4401e00a44e8f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41415 } } peers { permanent_uuid: "4463e65521a34a19a456696ada697046" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45645 } } peers { permanent_uuid: "7ede8d1d8476411a90984466081dc9cd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39319 } }
I20260924 04:18:10.668344 23280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ede8d1d8476411a90984466081dc9cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3405a778eb94bc19b4401e00a44e8f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41415 } } peers { permanent_uuid: "4463e65521a34a19a456696ada697046" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45645 } } peers { permanent_uuid: "7ede8d1d8476411a90984466081dc9cd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39319 } } }
I20260924 04:18:10.668543 23280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ede8d1d8476411a90984466081dc9cd [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:18:10.669262 23273 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ede8d1d8476411a90984466081dc9cd [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:18:10.670841 23199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:10.661975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39319 (local address 127.0.0.1:57904)
0924 04:18:10.662127 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 04:18:10.662140 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:18:10.662215 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:18:10.662568 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:18:10.662571 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:18:10.662746 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:18:10.662884 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:18:10.662892 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:18:10.663813 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:18:10.663816 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:18:10.664619 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:18:10.664629 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:18:10.665142 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:18:10.665145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:18:10.665223 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:10.665708 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:18:10.665722 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:18:10.666983 (+ 1261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:18:10.669161 (+ 2178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:18:10.669165 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:18:10.669167 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:18:10.669421 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:18:10.669742 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:18:10.669745 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:18:10.669747 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:18:10.669861 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:18:10.670330 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:18:10.670333 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:18:10.670438 (+ 105us) client_negotiation.cc:769] Sending connection context
0924 04:18:10.670534 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 04:18:10.670671 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260924 04:18:10.670992 23277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:10.662041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57904 (local address 127.0.98.60:39319)
0924 04:18:10.662359 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:10.662364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:10.662376 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:18:10.662410 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:10.662412 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:10.662451 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:10.662984 (+ 533us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:10.663060 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:10.663721 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:10.664720 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:10.665039 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:10.665136 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:10.667152 (+ 2016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:18:10.667156 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:18:10.667158 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:18:10.667180 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:18:10.669066 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:10.669521 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:10.669524 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:10.669526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:10.669606 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:10.669971 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:10.669974 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:10.669975 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:10.670189 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:18:10.670332 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 04:18:10.670698 (+ 366us) server_negotiation.cc:299] Negotiation successful
0924 04:18:10.670845 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":230}
I20260924 04:18:10.671496 23125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:10.662443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39319 (local address 127.0.0.1:57914)
0924 04:18:10.662817 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0924 04:18:10.662832 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:18:10.662903 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:18:10.663277 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:18:10.663280 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:18:10.663462 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:18:10.663557 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:18:10.663563 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:18:10.664564 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:18:10.664567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:18:10.665147 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:18:10.665157 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:18:10.665711 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:18:10.665715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:18:10.665799 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:10.666747 (+ 948us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:18:10.666758 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:18:10.668016 (+ 1258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:18:10.669942 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:18:10.669946 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:18:10.669948 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:18:10.670175 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:18:10.670430 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:18:10.670434 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:18:10.670435 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:18:10.670532 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:18:10.670971 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:18:10.670974 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:18:10.671082 (+ 108us) client_negotiation.cc:769] Sending connection context
0924 04:18:10.671174 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 04:18:10.671346 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305}
I20260924 04:18:10.673938 23197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4463e65521a34a19a456696ada697046: Bootstrap starting.
I20260924 04:18:10.675436 23279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:10.662592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57914 (local address 127.0.98.60:39319)
0924 04:18:10.662953 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:10.662984 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:10.663000 (+ 16us) server_negotiation.cc:407] Connection header received
0924 04:18:10.663055 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:10.663058 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:10.663120 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:10.663270 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:10.663668 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:10.664455 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:10.665239 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:10.665586 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:10.665679 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:10.668184 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:18:10.668188 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:18:10.668191 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:18:10.668216 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:18:10.669860 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:10.670255 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:10.670258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:10.670260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:10.670349 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:10.670609 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:10.670611 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:10.670613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:10.670835 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:18:10.670925 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 04:18:10.671948 (+ 1023us) server_negotiation.cc:299] Negotiation successful
0924 04:18:10.672060 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":272,"thread_start_us":94,"threads_started":1}
I20260924 04:18:10.676304 23197 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4463e65521a34a19a456696ada697046: Neither blocks nor log segments found. Creating new log.
I20260924 04:18:10.677160 23197 log.cc:824] T 00000000000000000000000000000000 P 4463e65521a34a19a456696ada697046: Log is configured to *not* fsync() on all Append() calls
I20260924 04:18:10.673170 23204 master_runner.cc:186] Error getting master registration for 127.0.98.62:41415: OK, instance_id { permanent_uuid: "b3405a778eb94bc19b4401e00a44e8f7" instance_seqno: 1790223490197047 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 04:18:10.678128 23204 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 04:18:10.678648 23122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3405a778eb94bc19b4401e00a44e8f7: Bootstrap starting.
I20260924 04:18:10.679409 23197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4463e65521a34a19a456696ada697046: No bootstrap required, opened a new log
W20260924 04:18:10.680858 23291 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7ede8d1d8476411a90984466081dc9cd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:18:10.680958 23291 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:18:10.681208 23122 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3405a778eb94bc19b4401e00a44e8f7: Neither blocks nor log segments found. Creating new log.
I20260924 04:18:10.682044 23122 log.cc:824] T 00000000000000000000000000000000 P b3405a778eb94bc19b4401e00a44e8f7: Log is configured to *not* fsync() on all Append() calls
I20260924 04:18:10.683452 23197 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4463e65521a34a19a456696ada697046 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3405a778eb94bc19b4401e00a44e8f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41415 } } peers { permanent_uuid: "4463e65521a34a19a456696ada697046" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45645 } } peers { permanent_uuid: "7ede8d1d8476411a90984466081dc9cd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39319 } }
I20260924 04:18:10.683681 23197 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4463e65521a34a19a456696ada697046 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4463e65521a34a19a456696ada697046, State: Initialized, Role: FOLLOWER
I20260924 04:18:10.684028 23197 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4463e65521a34a19a456696ada697046 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3405a778eb94bc19b4401e00a44e8f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41415 } } peers { permanent_uuid: "4463e65521a34a19a456696ada697046" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45645 } } peers { permanent_uuid: "7ede8d1d8476411a90984466081dc9cd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39319 } }
I20260924 04:18:10.684198 23122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3405a778eb94bc19b4401e00a44e8f7: No bootstrap required, opened a new log
I20260924 04:18:10.684645 23295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4463e65521a34a19a456696ada697046 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3405a778eb94bc19b4401e00a44e8f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41415 } } peers { permanent_uuid: "4463e65521a34a19a456696ada697046" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45645 } } peers { permanent_uuid: "7ede8d1d8476411a90984466081dc9cd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39319 } } }
I20260924 04:18:10.684991 23295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4463e65521a34a19a456696ada697046 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:18:10.685400 23197 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4463e65521a34a19a456696ada697046 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:18:10.687132 23122 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3405a778eb94bc19b4401e00a44e8f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3405a778eb94bc19b4401e00a44e8f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41415 } } peers { permanent_uuid: "4463e65521a34a19a456696ada697046" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45645 } } peers { permanent_uuid: "7ede8d1d8476411a90984466081dc9cd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39319 } }
I20260924 04:18:10.687381 23122 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3405a778eb94bc19b4401e00a44e8f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3405a778eb94bc19b4401e00a44e8f7, State: Initialized, Role: FOLLOWER
I20260924 04:18:10.687717 23122 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3405a778eb94bc19b4401e00a44e8f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3405a778eb94bc19b4401e00a44e8f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41415 } } peers { permanent_uuid: "4463e65521a34a19a456696ada697046" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45645 } } peers { permanent_uuid: "7ede8d1d8476411a90984466081dc9cd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39319 } }
W20260924 04:18:10.690629 23306 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4463e65521a34a19a456696ada697046: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:18:10.690701 23306 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:18:10.689149 23129 master_runner.cc:186] Error getting master registration for 127.0.98.62:41415: OK, instance_id { permanent_uuid: "b3405a778eb94bc19b4401e00a44e8f7" instance_seqno: 1790223490197047 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 04:18:10.692237 23129 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 04:18:10.692723 23122 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3405a778eb94bc19b4401e00a44e8f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:18:10.692785 23096 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3405a778eb94bc19b4401e00a44e8f7 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 04:18:10.692903 23096 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3405a778eb94bc19b4401e00a44e8f7 [term 0 FOLLOWER]: Advancing to term 1
W20260924 04:18:10.693689 23075 tablet.cc:2369] T 00000000000000000000000000000000 P b3405a778eb94bc19b4401e00a44e8f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:18:10.694126 23096 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3405a778eb94bc19b4401e00a44e8f7 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3405a778eb94bc19b4401e00a44e8f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41415 } } peers { permanent_uuid: "4463e65521a34a19a456696ada697046" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45645 } } peers { permanent_uuid: "7ede8d1d8476411a90984466081dc9cd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39319 } }
I20260924 04:18:10.695019 23096 leader_election.cc:288] T 00000000000000000000000000000000 P b3405a778eb94bc19b4401e00a44e8f7 [CANDIDATE]: Term 1 election: Requested vote from peers 4463e65521a34a19a456696ada697046 (127.0.98.61:45645), 7ede8d1d8476411a90984466081dc9cd (127.0.98.60:39319)
I20260924 04:18:10.695767 23247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b3405a778eb94bc19b4401e00a44e8f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7ede8d1d8476411a90984466081dc9cd"
I20260924 04:18:10.696022 23247 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ede8d1d8476411a90984466081dc9cd [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:18:10.696055 23171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b3405a778eb94bc19b4401e00a44e8f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4463e65521a34a19a456696ada697046"
I20260924 04:18:10.696269 23171 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4463e65521a34a19a456696ada697046 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:18:10.697111 23247 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7ede8d1d8476411a90984466081dc9cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3405a778eb94bc19b4401e00a44e8f7 in term 1.
I20260924 04:18:10.697453 23171 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4463e65521a34a19a456696ada697046 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3405a778eb94bc19b4401e00a44e8f7 in term 1.
I20260924 04:18:10.697536 23073 leader_election.cc:302] T 00000000000000000000000000000000 P b3405a778eb94bc19b4401e00a44e8f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ede8d1d8476411a90984466081dc9cd, b3405a778eb94bc19b4401e00a44e8f7; no voters:
W20260924 04:18:10.699836 23319 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b3405a778eb94bc19b4401e00a44e8f7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:18:10.699913 23319 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:18:10.700157 23307 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3405a778eb94bc19b4401e00a44e8f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3405a778eb94bc19b4401e00a44e8f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41415 } } peers { permanent_uuid: "4463e65521a34a19a456696ada697046" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45645 } } peers { permanent_uuid: "7ede8d1d8476411a90984466081dc9cd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39319 } } }
I20260924 04:18:10.700273 23307 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3405a778eb94bc19b4401e00a44e8f7 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:18:10.700356 23307 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3405a778eb94bc19b4401e00a44e8f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:18:10.700417 23307 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3405a778eb94bc19b4401e00a44e8f7 [term 1 LEADER]: Becoming Leader. State: Replica: b3405a778eb94bc19b4401e00a44e8f7, State: Running, Role: LEADER
I20260924 04:18:10.700698 23307 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3405a778eb94bc19b4401e00a44e8f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3405a778eb94bc19b4401e00a44e8f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41415 } } peers { permanent_uuid: "4463e65521a34a19a456696ada697046" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45645 } } peers { permanent_uuid: "7ede8d1d8476411a90984466081dc9cd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39319 } }
I20260924 04:18:10.702548 23054 master_runner.cc:263] No leader master found from master b3405a778eb94bc19b4401e00a44e8f7
I20260924 04:18:10.702612 23054 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 04:18:10.707645 23124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:10.697388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44692 (local address 127.0.98.62:41415)
0924 04:18:10.697496 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:10.697500 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:10.698553 (+ 1053us) server_negotiation.cc:407] Connection header received
0924 04:18:10.698715 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:10.698718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:10.698761 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:10.698807 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:10.699831 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:10.700613 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:10.701952 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:10.702280 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:10.702359 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:10.704197 (+ 1838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:18:10.704201 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:18:10.704204 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:18:10.704229 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:18:10.705839 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:10.706305 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:10.706309 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:10.706311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:10.706401 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:10.706671 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:10.706674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:10.706676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:10.706894 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:18:10.707006 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 04:18:10.707377 (+ 371us) server_negotiation.cc:299] Negotiation successful
0924 04:18:10.707493 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 04:18:10.714702 23324 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3405a778eb94bc19b4401e00a44e8f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3405a778eb94bc19b4401e00a44e8f7. Latest consensus state: current_term: 1 leader_uuid: "b3405a778eb94bc19b4401e00a44e8f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3405a778eb94bc19b4401e00a44e8f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41415 } } peers { permanent_uuid: "4463e65521a34a19a456696ada697046" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45645 } } peers { permanent_uuid: "7ede8d1d8476411a90984466081dc9cd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39319 } } }
I20260924 04:18:10.714859 23324 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3405a778eb94bc19b4401e00a44e8f7 [sys.catalog]: This master's current role is: LEADER
I20260924 04:18:10.717399 23279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:10.698124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57920 (local address 127.0.98.60:39319)
0924 04:18:10.698232 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:10.698236 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:10.707673 (+ 9437us) server_negotiation.cc:407] Connection header received
0924 04:18:10.707877 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:10.707881 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:10.707934 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:10.707981 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:10.709377 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:10.710181 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:10.711428 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:10.711792 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:10.711863 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:10.713865 (+ 2002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:18:10.713869 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:18:10.713872 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:18:10.713899 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:18:10.715638 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:10.716100 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:10.716104 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:10.716106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:10.716185 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:10.716466 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:10.716469 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:10.716471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:10.716669 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:18:10.716776 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 04:18:10.717105 (+ 329us) server_negotiation.cc:299] Negotiation successful
0924 04:18:10.717225 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 04:18:10.718711 23330 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:18:10.719691 23330 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:18:10.723892 23203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:10.711599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35812 (local address 127.0.98.61:45645)
0924 04:18:10.711715 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:10.711719 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:10.713930 (+ 2211us) server_negotiation.cc:407] Connection header received
0924 04:18:10.713982 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:10.713984 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:10.714034 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:10.714111 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:10.715795 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:10.716580 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:10.717672 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:10.718006 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:10.718075 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:10.720334 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:18:10.720338 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:18:10.720341 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:18:10.720370 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:18:10.722042 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:10.722570 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:10.722573 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:10.722575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:10.722662 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:18:10.722936 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:18:10.722939 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:18:10.722941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:18:10.723152 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:18:10.723281 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 04:18:10.723616 (+ 335us) server_negotiation.cc:299] Negotiation successful
0924 04:18:10.723735 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 04:18:10.732924 23247 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7ede8d1d8476411a90984466081dc9cd [term 1 FOLLOWER]: Refusing update from remote peer b3405a778eb94bc19b4401e00a44e8f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:18:10.733678 23307 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b3405a778eb94bc19b4401e00a44e8f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ede8d1d8476411a90984466081dc9cd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:18:10.733922 23171 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4463e65521a34a19a456696ada697046 [term 1 FOLLOWER]: Refusing update from remote peer b3405a778eb94bc19b4401e00a44e8f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:18:10.734447 23307 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b3405a778eb94bc19b4401e00a44e8f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4463e65521a34a19a456696ada697046" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:18:10.738054 23295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4463e65521a34a19a456696ada697046 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3405a778eb94bc19b4401e00a44e8f7. Latest consensus state: current_term: 1 leader_uuid: "b3405a778eb94bc19b4401e00a44e8f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3405a778eb94bc19b4401e00a44e8f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41415 } } peers { permanent_uuid: "4463e65521a34a19a456696ada697046" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45645 } } peers { permanent_uuid: "7ede8d1d8476411a90984466081dc9cd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39319 } } }
I20260924 04:18:10.738179 23295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4463e65521a34a19a456696ada697046 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:18:10.738930 23280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ede8d1d8476411a90984466081dc9cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3405a778eb94bc19b4401e00a44e8f7. Latest consensus state: current_term: 1 leader_uuid: "b3405a778eb94bc19b4401e00a44e8f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3405a778eb94bc19b4401e00a44e8f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41415 } } peers { permanent_uuid: "4463e65521a34a19a456696ada697046" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45645 } } peers { permanent_uuid: "7ede8d1d8476411a90984466081dc9cd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39319 } } }
I20260924 04:18:10.739051 23280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ede8d1d8476411a90984466081dc9cd [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:18:10.744088 23324 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3405a778eb94bc19b4401e00a44e8f7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b3405a778eb94bc19b4401e00a44e8f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3405a778eb94bc19b4401e00a44e8f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41415 } } peers { permanent_uuid: "4463e65521a34a19a456696ada697046" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45645 } } peers { permanent_uuid: "7ede8d1d8476411a90984466081dc9cd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39319 } } }
I20260924 04:18:10.744217 23324 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3405a778eb94bc19b4401e00a44e8f7 [sys.catalog]: This master's current role is: LEADER
I20260924 04:18:10.744319 23331 mvcc.cc:205] Tried to move back new op lower bound from 7332755418010357760 to 7332755417956442112. Current Snapshot: MvccSnapshot[applied={T|T < 7332755418010357760}]
I20260924 04:18:10.744390 23295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4463e65521a34a19a456696ada697046 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b3405a778eb94bc19b4401e00a44e8f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3405a778eb94bc19b4401e00a44e8f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41415 } } peers { permanent_uuid: "4463e65521a34a19a456696ada697046" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45645 } } peers { permanent_uuid: "7ede8d1d8476411a90984466081dc9cd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39319 } } }
I20260924 04:18:10.744496 23295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4463e65521a34a19a456696ada697046 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:18:10.746085 23307 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3405a778eb94bc19b4401e00a44e8f7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b3405a778eb94bc19b4401e00a44e8f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3405a778eb94bc19b4401e00a44e8f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41415 } } peers { permanent_uuid: "4463e65521a34a19a456696ada697046" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45645 } } peers { permanent_uuid: "7ede8d1d8476411a90984466081dc9cd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39319 } } }
I20260924 04:18:10.746201 23307 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3405a778eb94bc19b4401e00a44e8f7 [sys.catalog]: This master's current role is: LEADER
I20260924 04:18:10.747169 23280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ede8d1d8476411a90984466081dc9cd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b3405a778eb94bc19b4401e00a44e8f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3405a778eb94bc19b4401e00a44e8f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41415 } } peers { permanent_uuid: "4463e65521a34a19a456696ada697046" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45645 } } peers { permanent_uuid: "7ede8d1d8476411a90984466081dc9cd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39319 } } }
I20260924 04:18:10.747321 23280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ede8d1d8476411a90984466081dc9cd [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:18:10.748126 23333 mvcc.cc:205] Tried to move back new op lower bound from 7332755418010357760 to 7332755417956442112. Current Snapshot: MvccSnapshot[applied={T|T < 7332755418010357760}]
I20260924 04:18:10.751600 23332 mvcc.cc:205] Tried to move back new op lower bound from 7332755418010357760 to 7332755417956442112. Current Snapshot: MvccSnapshot[applied={T|T < 7332755418010357760}]
I20260924 04:18:10.752193 23330 catalog_manager.cc:1347] Generated new cluster ID: de1455108302478c898911428f652b77
I20260924 04:18:10.752255 23330 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:18:10.770105 23330 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:18:10.771474 23330 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:18:10.801429 23330 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3405a778eb94bc19b4401e00a44e8f7: Generated new TSK 0
I20260924 04:18:10.802558 23330 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 04:18:10.833038 23279 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:18:10.829877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57924 (local address 127.0.98.60:39319)
0924 04:18:10.830035 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:10.830040 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:10.831894 (+ 1854us) server_negotiation.cc:407] Connection header received
0924 04:18:10.832119 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:10.832123 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:10.832172 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:10.832224 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:10.832897 (+ 673us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57924: BlockingRecv error: recv got EOF from 127.0.0.1:57924 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260924 04:18:10.834421 23124 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:18:10.830614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44700 (local address 127.0.98.62:41415)
0924 04:18:10.830728 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:10.830732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:10.833526 (+ 2794us) server_negotiation.cc:407] Connection header received
0924 04:18:10.833688 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:10.833692 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:10.833739 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:10.833787 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:10.834295 (+ 508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44700: BlockingRecv error: recv got EOF from 127.0.0.1:44700 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260924 04:18:10.835732 23203 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:18:10.831304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35820 (local address 127.0.98.61:45645)
0924 04:18:10.831423 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:10.831427 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:10.834867 (+ 3440us) server_negotiation.cc:407] Connection header received
0924 04:18:10.834975 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:10.834978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:10.835028 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:10.835100 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:10.835619 (+ 519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35820: BlockingRecv error: recv got EOF from 127.0.0.1:35820 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260924 04:18:10.845145 23203 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:18:10.843691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35834 (local address 127.0.98.61:45645)
0924 04:18:10.843814 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:10.843818 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:10.844079 (+ 261us) server_negotiation.cc:407] Connection header received
0924 04:18:10.844314 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:10.844318 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:10.844401 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:10.844458 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:10.845021 (+ 563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35834: BlockingRecv error: recv got EOF from 127.0.0.1:35834 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260924 04:18:10.850509 23203 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:18:10.848778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35842 (local address 127.0.98.61:45645)
0924 04:18:10.848893 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:10.848898 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:10.849371 (+ 473us) server_negotiation.cc:407] Connection header received
0924 04:18:10.849765 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:10.849769 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:10.849824 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:10.849886 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:10.850404 (+ 518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35842: BlockingRecv error: recv got EOF from 127.0.0.1:35842 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260924 04:18:10.856683 23203 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:18:10.854967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35852 (local address 127.0.98.61:45645)
0924 04:18:10.855117 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:10.855121 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:10.855672 (+ 551us) server_negotiation.cc:407] Connection header received
0924 04:18:10.855929 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:10.855932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:10.855986 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:10.856036 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:10.856573 (+ 537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35852: BlockingRecv error: recv got EOF from 127.0.0.1:35852 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260924 04:18:10.865002 23203 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:18:10.863344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35864 (local address 127.0.98.61:45645)
0924 04:18:10.863536 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:10.863541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:10.864062 (+ 521us) server_negotiation.cc:407] Connection header received
0924 04:18:10.864269 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:10.864272 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:10.864322 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:10.864373 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:10.864893 (+ 520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35864: BlockingRecv error: recv got EOF from 127.0.0.1:35864 (error 108)
Metrics: {"server-negotiator.queue_time_us":122}
W20260924 04:18:10.877322 23203 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:18:10.875577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35880 (local address 127.0.98.61:45645)
0924 04:18:10.875702 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:10.875706 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:10.876271 (+ 565us) server_negotiation.cc:407] Connection header received
0924 04:18:10.876490 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:10.876494 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:10.876553 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:10.876609 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:10.877184 (+ 575us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35880: BlockingRecv error: recv got EOF from 127.0.0.1:35880 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260924 04:18:10.897485 23203 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:18:10.896127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35882 (local address 127.0.98.61:45645)
0924 04:18:10.896263 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:10.896269 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:10.896492 (+ 223us) server_negotiation.cc:407] Connection header received
0924 04:18:10.896620 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:10.896624 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:10.896695 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:10.896807 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:10.897335 (+ 528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35882: BlockingRecv error: recv got EOF from 127.0.0.1:35882 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260924 04:18:10.932472 23203 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:18:10.931362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35886 (local address 127.0.98.61:45645)
0924 04:18:10.931507 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:10.931512 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:10.931634 (+ 122us) server_negotiation.cc:407] Connection header received
0924 04:18:10.931780 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:10.931784 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:10.931836 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:10.931904 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:10.932349 (+ 445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35886: BlockingRecv error: recv got EOF from 127.0.0.1:35886 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260924 04:18:10.999570 23203 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:18:10.998322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35902 (local address 127.0.98.61:45645)
0924 04:18:10.998515 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:10.998519 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:10.998659 (+ 140us) server_negotiation.cc:407] Connection header received
0924 04:18:10.998830 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:10.998833 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:10.998882 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:10.998953 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:10.999447 (+ 494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35902: BlockingRecv error: recv got EOF from 127.0.0.1:35902 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260924 04:18:11.130488 23203 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:18:11.129477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35916 (local address 127.0.98.61:45645)
0924 04:18:11.129619 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:11.129624 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:11.129753 (+ 129us) server_negotiation.cc:407] Connection header received
0924 04:18:11.129864 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:11.129866 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:11.129916 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:11.129986 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:11.130364 (+ 378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35916: BlockingRecv error: recv got EOF from 127.0.0.1:35916 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260924 04:18:11.389626 23203 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:18:11.388578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35924 (local address 127.0.98.61:45645)
0924 04:18:11.388720 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:11.388725 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:11.388758 (+ 33us) server_negotiation.cc:407] Connection header received
0924 04:18:11.388921 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:11.388924 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:11.388973 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:11.389040 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:18:11.389507 (+ 467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35924: BlockingRecv error: recv got EOF from 127.0.0.1:35924 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 04:18:11.682319 23291 catalog_manager.cc:1259] Loaded cluster ID: de1455108302478c898911428f652b77
I20260924 04:18:11.682406 23291 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7ede8d1d8476411a90984466081dc9cd: loading cluster ID for follower catalog manager: success
I20260924 04:18:11.684213 23291 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7ede8d1d8476411a90984466081dc9cd: acquiring CA information for follower catalog manager: success
I20260924 04:18:11.685395 23291 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7ede8d1d8476411a90984466081dc9cd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 04:18:11.691830 23306 catalog_manager.cc:1259] Loaded cluster ID: de1455108302478c898911428f652b77
I20260924 04:18:11.691905 23306 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4463e65521a34a19a456696ada697046: loading cluster ID for follower catalog manager: success
I20260924 04:18:11.693535 23306 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4463e65521a34a19a456696ada697046: acquiring CA information for follower catalog manager: success
I20260924 04:18:11.694621 23306 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4463e65521a34a19a456696ada697046: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 04:18:11.907174 23406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:11.903520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35926 (local address 127.0.98.61:45645)
0924 04:18:11.903747 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:11.903751 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:11.903767 (+ 16us) server_negotiation.cc:407] Connection header received
0924 04:18:11.903925 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:11.903928 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:11.904050 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:11.904145 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:18:11.904419 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:11.904945 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:11.905774 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:11.906005 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:11.906091 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:11.906378 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:18:11.906792 (+ 414us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:18:11.906904 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 04:18:11.906965 (+ 61us) server_negotiation.cc:299] Negotiation successful
0924 04:18:11.907021 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":65,"threads_started":1}
I20260924 04:18:11.913029 23412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:11.909882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57932 (local address 127.0.98.60:39319)
0924 04:18:11.910066 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:11.910069 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:11.910081 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:18:11.910205 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:11.910207 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:11.910313 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:11.910427 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:18:11.910559 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:11.911037 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:11.911813 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:11.912040 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:11.912113 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:11.912322 (+ 209us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:18:11.912730 (+ 408us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:18:11.912815 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 04:18:11.912841 (+ 26us) server_negotiation.cc:299] Negotiation successful
0924 04:18:11.912886 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":57,"threads_started":1}
I20260924 04:18:11.919412 23406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:11.915824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35942 (local address 127.0.98.61:45645)
0924 04:18:11.915944 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:11.915949 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:11.916032 (+ 83us) server_negotiation.cc:407] Connection header received
0924 04:18:11.916213 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:11.916217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:11.916264 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:11.916308 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:18:11.916628 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:11.917389 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:11.918242 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:11.918504 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:11.918592 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:11.918922 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:18:11.919005 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:18:11.919073 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 04:18:11.919202 (+ 129us) server_negotiation.cc:299] Negotiation successful
0924 04:18:11.919280 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 04:18:11.919492 23412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:11.915818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57942 (local address 127.0.98.60:39319)
0924 04:18:11.915925 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:11.915929 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:11.916371 (+ 442us) server_negotiation.cc:407] Connection header received
0924 04:18:11.916447 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:11.916450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:11.916489 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:11.916542 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:18:11.916843 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:11.917314 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:11.918416 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:11.918704 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:11.918793 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:11.919059 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:18:11.919132 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:18:11.919188 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 04:18:11.919350 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 04:18:11.919400 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 04:18:11.919971 23419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:18:11.915802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44708 (local address 127.0.98.62:41415)
0924 04:18:11.916176 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0924 04:18:11.916181 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:18:11.916193 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:18:11.916277 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:18:11.916280 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:18:11.916445 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:18:11.916560 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:18:11.916771 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:11.917543 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:11.918584 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:18:11.918853 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:18:11.918953 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:18:11.919215 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:18:11.919569 (+ 354us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:18:11.919654 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 04:18:11.919780 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 04:18:11.919831 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":79,"threads_started":1}
I20260924 04:18:11.922749 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 23054
I20260924 04:18:11.928437 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 23129
I20260924 04:18:11.934149 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6HV92s/build/debug/bin/kudu with pid 23204
2026-09-24T04:18:11Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2998 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5970 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (275658 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST